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ABSTRACT 



Policy interest in international surveys on Adult Learning (AL) has increased strongly. AL survey 
data are used as benchmarks for a country’s educational system. However, results of key indicators like 
participation in learning activities often vary remarkably between different data sources. Stating that these 
differences are due to varying concepts and methods is not enough. The key question is: Which figures 
represent reality more appropriately? Therefore, evaluation of survey concepts and methods is crucial for 
international comparison of Adult Learning. 

This report provides guidelines on methodological and conceptual issues. Part one covers 
methodological aspects while part two deals with concepts, definitions and example questions. 
Recommendations are based on input from 14 countries. 

The methodological section covers data collection (telephone vs. f2f-interviews and online surveys); 
reference period (calendar year; recall problems); target population (age limit; non-national residents); 
sample design (proxy interviews, random selection of learning activities). 

The second part first discusses the basic concepts of Adult learning used in the European Adult 
Education Survey and in non-European countries (e.g. Canada, USA), including informal learning. The 
report then goes on to discuss empirical concepts and questions on AL participation and shows how 
different concepts affect empirical results and recommends example questions for formal, non-formal and 
informal learning. Other chapters refer to volume of AL; costs (what can be answered by non-experts?); 
providers of AL (define by teacher or learning location?); fields (ISCED, etc.); obstacles and benefits 
(which items are more valid?) and imputation. 

This report will act as a useful resource tool for researchers and policy makers when designing new 
national AL surveys or when optimising existing surveys. Results are focussed by a summary of 
conclusions, recommendations and example questions at the end of each section. 
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RESUME 



L’interet politique envers les enquetes portant sur la formation des adultes s’est fortement accru. Les 
donnees issues des etudes menees sur ce theme sont utilisees comme mesures de la performance des 
systemes educatifs nationaux. Pourtant, les resultats d’indicateurs cles tels que la participation aux activites 
de formation varient sou vent de fa 9 on marquee selon la source des donnees. II ne suffit cependant pas 
d’invoquer les divers concepts et methodes employes pour expliquer ces variations. En effet, la question 
cle est la suivante : quels sont les chiffres qui representent la realite de la fa^on la plus appropriee ? Par 
consequent, 1’evaluation des concepts et methodes d’enquete s’avere cruciale pour la comparaison 
internationale des formations des adultes. 

Le present rapport founit des directives sur des questions methodologiques et conceptuelles. La 
premiere partie traite des aspects methodologiques, tandis que la deuxieme partie presente les concepts, 
definitions et les exemples de questions. Les recommandations formulees reposent sur des donnees 
foumies par 14 pays. 

La section methodologie couvre la collecte de donnees (par telephone, par entretiens en personne et 
par enquetes sur Internet) ; la periode de reference (annee civile, difficulte des repondants a faire appel a 
leur memoire) ; la population ciblee (limite d’age, residents non-ressortissants) ; et le plan 
d’echantillonnage (entretien par personne interposee, selection au hasard des activites de formation). 

La seconde partie dehat des concepts de base utilises dans I’Enquete europeenne sur la formation des 
adultes (European Adult Education Survey) et dans le cadre d’autres enquetes non-europeennes (par 
exemple canadiennes ou americaines), y compris Eapprentissage informel. Le rapport ahorde ensuite les 
concepts empiriques et les questions qui concernent la participation a la formation des adultes. Les autres 
chapitres se referent au volume de formation des adultes ; aux couts (quelles reponses peuvent-elles etre 
apportees par des non experts ?) ; aux fournisseurs (doit-on les definir selon les enseignants ou selon le 
lieu des cours ?) ; aux domaines (CITE, etc.) ; aux obstacles et aux avantages (quels elements sont-ils les 
plus valides ?) et a Eimputation. 

Le present rapport servira d’outil aux chercheurs et aux decideurs dans 1’ elaboration de nouvelles 
enquetes nationales sur la formation des adultes ou hien dans les efforts d’ amelioration des enquetes 
existantes. Les resultats sont presentes de fa 9 on cihlee dans un resume des conclusions, des 
recommandations et des exemples de questions a la fin de chaque section. 
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1. Preface: Development of an International Adult Learning Module (OECD AL Module) - 
objectives and concept 

Objectives, concepts and starting point of the OECD Adult Learning module (OECD AL module) are 
stated very clearly in the following text taken from the terms of reference of the project: 

“In 2001... Network decided to tackle the severe comparability issues of statistics on continuing 
education and training through a project called the Network B CET Module Project. After obtaining 
financial support from 14 countries, Doug Giddings and Helmut Kuwan prepared a report under the 
guidance of the participating countries. The main objective of the project was to develop a set of guidelines 
for the development of internationally comparable indicators on CET (continuing education and training)^ 
to be based on data collected via household surveys.. . . 

At its meeting in Helsinki, in May 2005, the CET Working Group and the plenary session of Network 
B approved the decision to produce a Network B final report on the CET Module activity. . . The document 
should clearly state the various parameters and content issues for the development of internationally 
comparable statistics on CET or LLL (life-long-leaming)... 

The report will take the consultants’ final report of the Network B CET Module Project as a starting 
point (in which many issues about concepts, definitions and methodologies have been examined). Since 
further developments have taken place internationally, especially with the preparation of the European 
Adult Education Survey (AES), it is also necessary for the report to integrate the work done in the context 
of the preparation of the AES (and possibly the lessons from pilot surveys in some countries) and other 
relevant recent documents such as the Classification of Learning Activities.” 

As agreed, the report should provide guidelines for the development of surveys on continuing 
education and training/lifelong learning or adult learning (AL), which would produce internationally 
comparable data based on accepted common concepts, definitions and methodologies. Therefore, it should 
cover the following issues: 

Proposed definitions of learning, e.g. formal education, non-formal education and informal learning; 
the key topic areas to be covered by a survey on adult learning and proposed methods; a review of 
methodological issues and their proposed resolution; the identification of the characteristics of a national 
“carrier” survey, for the cases when national surveys are run outside the context of an international project; 
the development of interpretative guidelines for items of the AL module. 

The issues discussed here are not only important for statistical experts, they are also important for 
educational policy development in general, especially when results are used as benchmarks for a country’s 
educational system. Results of key indicators like participation in learning activities often vary between 
data sources, and sometimes these differences are quite remarkable. Picture 1 illustrates this problem in an 
example based on real data sources. 



^ Network B develops indicators of social and economic outcomes of education for the OECD’s project 
International Indicators of Education Systems (INES) The INES Project responds to the needs of countries for 
information and indicators that allow comparisons of the performance of education systems and better assessment and 
monitoring of the effectiveness of education systems. 



When used first acronyms are spelled out in the text. Furthermore, there is a list of acronyms in Annex 1 . 
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Figure 1: 

Participation in non-formal non-vocational learning according to different data sources 

2003 - Country A 




National 
data source 



0,5 million 
participants 



International 
data source 




Integrated provider 
statistics 



[> Which figure represents reality more appropriately? 



Helmut Kuwait 2008, OECD AL Module 



This example may illustrate why the dimension of the problem is not merely a statistical one. It is 
very likely that different results will evoke different conclusions for educational policy. Therefore, 
statistical discussions should go heyond the point of stating that differences are due to varying concepts 
and methods and try to find clues for a rating of concepts and methods. When results vary strongly 
between different data sources, one of the key questions^ from a political point of view simply is: Which 
figure represents reality more appropriately? 

Varying results between different surveys may be caused by: 

• Methodological differences in design or realisation of a survey; 

• The use of varying concepts and definitions. 

Accordingly, the report consists of two main parts. Part one covers methodological aspects while part 
two deals with concepts, definitions and example questions. Two aspects are particularly important: 

• The report aims at recommendations, not descriptions. It tries to state which concepts are better 
than others and why; 

• The report deals with content issues and methodological issues. 



Other important questions for policy makers are: What is the policy question that needs to be addressed? What 
measure best approximates information that is needed to answer the question? Are existing measures 
sufficiently accurate and reliable to answer the question? 

In this example, integrated provider statistics indicate that this is the national data source, even though the 
indicators used in the data sources are not strictly the same (participants and participation cases). 
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At the end of each section of this report, a graph or a short summary is given (“conclusions”). This 
summary addresses important conclusions and recommendations referring to the issues addressed in the 
chapter. 

This report not only follows the work done by Doug Giddings and Helmut Kuwan, it also uses input 
from Ann-Charlotte Larsson from Statistics Sweden. Very helpful input to this report was given by Alistair 
Nolan, Bo Hansson and Fionnuala Canning (OECD), Christiane Kriiger-Hemmer (Statistical Office 
Germany), Patrice de Broucker (Statistics Canada), Lisa Hudson (US Department of Education) and the e- 
mail discussion group of the financing countries. Special thanks go to Chris Chapman (US Department of 
Education) for a very inspiring review of the report and to Andrea Reupold (University of Munich) for 
investigating and structuring a wide range of material. 

Following developments in recent years, the title of this report was changed from OECD CET Module 
to OECD Adult Learning Module (OECD AL module). 

The author of the OECD AL module report is Helmut Kuwan. 
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2. Priority issues 

The decisions on the priority of topics in an AL survey should also take into account policy interest 
and statistical feasihility. In phase 1 of the OECD-module -project, much effort was put on identifying 
priority issues (see Giddings / Kuwan 2002, p. 25ff.) Decisions made in the EU AES and feedback from 
members of INES Network B helped to update these priorities. 

Based on this information a proposal for a set of priority issues was presented at the Network B 
Plenary meeting in Washington on March 7*, 2006. The proposal included a design for a core and an 
additional part of the OECD AL module and presented priority issues for both parts of the module. 

In Washington, members of Network B supported the idea of designing a core and an additional part 
of the OECD module. All priority ratings in the core module and most ratings of the additional module 
were confirmed. 



Figure 2: 

Core issues of the OECD AL module 

★ Participation in learning activities (formal, non-formal, informal) 

★ Purpose (job-related and personal interest) 

★ Volume (formal and non-formal learning only) 

★ Costs (formal and non-formal learning only) 

★ Fields of learning 

★ Providers 

★ Obstacles and incentives 

Helmut Kuwan 2008, OECD AL Module 2 
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Figure 3: 

Examples of additional priority issues in an OECD AL module 

In priority order of examination 

★ Possibly record for each activity: 

► Output, credentials 

► Quality (satisfaction?) 

► Methods of delivery (including distance learning) 

► [Skills - self-report] 

★ Outcomes 

★ Changes in the workplace 

★ Company's learning environment: Encouraging or inhibitory? 

★ Learning preferences 

★ Private learning environment 



Helmut Kuwan 2008, OECD AL Module 3 



3. Methodological aspects 
3.1 Method of data collection 

Various methods of gathering data are used in AL surveys including face-to-face interviews (f2f- 
interviews), telephone interviews and mail surveys. More recently, online-surveys have increased. 
Determining the method that will yield the most valid results is a fundamental issue when designing 
international AL-surveys. 

At first sight, online-surveys seem to he an attractive option because of their low costs hut presently 
they will not lead to representative population samples in any of the countries concerned.^ The reason for 
this is simply that a considerable part of the population does not use the internet. Internet users differ 
significantly from the total adult population with regard to a variety of important variables like age, 
educational level, etc. 

Designing a stratified online sample in which the stratified criteria represent exactly the total 
population does not solve this problem, nor would ex-post-weighing procedures because variables not 
included in the stratifying or weighing procedures still may differ considerably. In the long-run online- 
surveys may become an interesting option, if the structure of online samples and the total population 
become very similar and other criteria are also met ^ 

In mail interviews, it is not possible to control whether the target person or another person fills in the 
questionnaire. Therefore, only telephone-interviews and face-to-face-interviews (f2f-interviews) are 
considered serious options for an AL survey. 



With regard to company samples or specific “internet-related” sub-groups this may be different. 

In addition to the structures of the online users and those of the population, response rates and data validity of 
online-surveys have to be checked, too. 
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One necessary condition for conducting a representative telephone survey is that the share of 
households without a telephone is “small”. For an international AL Survey conducted hy telephone it 
would he helpful to define a cut-off limit stating the share of households without a telephone in the 
population helow which this method is unacceptable. Methodologically, no exact figure for a cut-off limit 
can he fixed. The author considers a rate of 5% or less of households without a telephone to he acceptable 
whereas a rate of 10% or higher would be too high. 

Random digit dialling requires a conventional telephone network. If cellular phones increasingly 
become the only telephone extension of a household, this may seriously affect the validity of data from 
telephone interviews.^ 

One main difference between telephone and f2f-interviews is the possibility to give visual support in 
f2f-interviews to help respondents to answer complex questions. Although this is not possible in telephone 
interviews, it is generally agreed that most questions and answers can be “translated”. So far, no 
information is available on whether slightly different wordings in telephone and f2f-interviews affect the 
results, but it is known from pre-tests and cognitive research that minor changes in wording may make a 
difference. 

In interviews, experts from survey institutes sometimes refer to unpublished knowledge based on 
practical experience (“tacit knowledge”).* An effect based on tacit knowledge reported to the author is that 
there may be an influence on the response rates and the results of a telephone survey if a direct reference to 
the AL-topic is given to the household already in the contact phase of the interview. This reference in the 
introduction tends to encourage active learners to participate in the interview and to discourage people with 
few or no learning activities. This effect can easily be avoided by describing the survey topic in a more 
general way. 

There is a general assumption that both of these interview methods lead to the same results. Is this 
assumption correct or are there systematic differences in results between the two methods? 

This is a fundamental question. Studies in Finland, Sweden and Germany reveal some differences 
based on which method was used,^ but apart from this, only limited information is available. 

Overall, however, evidence supporting the assumption of differing AL -results in f2f- and telephone 
interviews recently has increased, although no general pattern emerges. Tacit knowledge in Germany 
indicates that long lists of learning activities lead to higher participation rates in telephone than in f2f- 
interviews due to the different interview situation. In face-to-face interviews, the target person may look at 
an activity list whereas in a telephone interview every item has to be read by the interviewer. This might be 
a reason why results varied between a representative f2f-survey and a telephone survey in Germany 
although very similar AL -concepts were used. 

In contrast, participation rates in f2f-interviews in Sweden were higher than in telephone interviews 
but it was not clear if this was a mode effect. 



If valid samples of cellular phones were available, this problem would be solved. 

"Tacit Knowledge" is an expression for knowledge from practical experts that is not published and has not yet 
reached the scientific community. Often these issues become a topic in discussions of scientists with some 
delay, sometimes they remain tacit. Particularly when results are quite sensitive the information source 
often does not like to be quoted. 

See Kangassalo, Perth and Heiskanen, Markku, Testing the mode effects in the Finnish consumer survey, 2002. 
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Examples of results that correspond well in f2f- and telephone interviews exist too. Highly similar 
results for most of the major variables were found in Canada. 

These results illustrate the need for further research on the scope of differences and on the factors that 
explain the origin of these differences. They also show that the general assumption that face-to-face and 
telephone interviews in the area of AL lead to the same results does not seem to he certain. Although 
present information tends to favour the validity of f2f-results slightly, there is no clear empirical evidence 
to support which method is “hetter”. This question should he a focus of further methodological research. 

Last hut not least, survey costs will also he an important criterion when choosing the method of data 
collection. Although these costs vary with regard to parameters like interview length, telephone charges or 
travel expenses, in most countries the costs of population surveys hy telephone interviews will he lower 
than hy f2f-interviews. 

Conclusions 

1) Presently, online-surveys or mail surveys will not lead to representative data in international 
population surveys on AL. 

2) Eace-to-face and telephone interviews on adult learning activities may lead to different results. 
There is no clear evidence which method is hetter. 

3) To avoid possible method effects an international AL Survey should try to implement the same 
data collection method in all participating countries. If this is not possible analyses of 
international differences also ought to show a comparison between countries using the same data 
collection methods. 

Computer- Aided-Interviews and Paper-and-Pencil-Interviews 

In survey research the use of computer programs in controlling and assisting interviews has increased 
greatly. Computer Assisted Telephone Interview (CATI) and Computer Assisted Personal Interview 
(CAPI) are widely used. Still some countries seem to prefer paper-and-pencil-interviews. Therefore it is 
important to know if the results differ between computer assisted interviews and paper and pencil 
interviews. 

Little information is available on this issue. The main effect of using computer-assisted interviews 
seems to be the possibility of improved data quality. Some mistakes and inconsistencies can be identified 
already during the interview situation and corrected by the respondent instead of using ex -post data checks. 
The possibility of including automatic skips in order to avoid errors regarding the routing between 
questions is another advantage. A third important effect is a shortened post-interview time schedule. 

It is important to acknowledge, however, that specifications of the computer program may affect the 
results. To ensure comparability, the computer programs used in different countries should be identical or 
at least very similar. The programs have to be checked very carefully as mistakes will lead to a loss of 
information. 



In Canada the International Adult Literacy Survey (lALS) was conducted in f2f-Interviews about the same time 
as an Adult Education and Training Survey by telephone interviews which was built largely on the AETS 
questionnaire. Differing results for course duration were likely due to the lALS practice of including only 
the most recent courses while AETS collected information on all courses. 



12 




EDUAVKP(2008)8 



Conclusions 

1) Computer-Aided interviews potentially improve data quality. 

2) If possible, computer-aided interviews with identical or very similar programs should be used in an 
international AL Survey. 



Figure 4: 

Data collection methods in international population 
surveys on Adult Learning 



Do 

★ Face-to-face interviews or 
telephone interviews 




★ Try to implement the same 
method in different countries 



Do not 

★ Online surveys 

★ Mail surveys 




► Presently online surveys will not lead to representative data. 

► "Tacit knowledge" indicates that f2f- or telephone interviews may lead to different 
results. There is no clear evidence which method is better. 

► Therefore international AL surveys should he based on the same method in 
all participating countries. 



Helmut Kuwan 2008, OECD AL Module 



3.2 Reference period 

The reference period is a very important issue for an international AL Survey as this influences results 
relating to key indicators like participation in learning activities or volume. Therefore, advantages and 
disadvantages of different reference periods have been discussed for a long time. Most countries carrying 
out AL surveys use an annual reference period. Recent discussions mainly refer to three aspects: 

1) Should the reference period be one year or shorter? 

2) Should an annual reference period cover the calendar year or the last 12 months? 

3) Should a longer reference period than one year also be used? 

Should the reference period of an AL survey be one year or shorter? 

According to results from Statistics Sweden, a shorter reference period leads to a better recall of 
learning activities in non-formal education. On the other hand, serious disadvantages with regard to the 
preferences of statistical users and analytical limitations are connected with a shorter reference period: 
Most users are interested in estimates for a calendar year. Possible limitations of analytical potential are 
due to the fact that a shorter reference period, e.g. four weeks, probably will lead to rather low participation 



13 




EDU/WKP(2008)8 



rates in formal and non-formal education in many countries. If most individuals have not participated in 
any activity the potential for analysing differences between groups is limited very much. 

In addition, short reference periods may also he strongly influenced hy seasonal effects, whereas a 
reference period of one calendar year would avoid this bias. For these reasons, it is recommended here that 
a reference period of one year is used for measuring formal and non-formal education activities. In the EU 
AES, most countries use a reference period of the less than last 12 months from the time of the interview, 
or longer. 

As for informal learning, the situation is somewhat different. Since participation in informal learning 
usually is higher than in formal or non-formal education, the problem of limiting analytical potential by 
using a shorter reference period is less severe. In addition, recall problems seem to be bigger for informal 
learning. Therefore, adding a shorter reference period than 12 months, for example the last four weeks, 
would probably improve the quality of results on participation in informal learning activities. However, 
seasonal effects will occur in informal learning, too. 

Calendar year or the last 12 months? 

A reference period of one year could be a calendar year, the last 12 months before the interview, or 
some other fixed 12-month period. In existing surveys either the last 12 months from the time of the 
interview or the calendar year are used. Both options have advantages and disadvantages. 

Statistical users clearly prefer a calendar year. A calendar year also seems to correspond better with 
the time schedules of respondents than a period of the last 12 months." To give an example: It is easier for 
the respondent to think of activities in the year 2006 than of activities from September 1 2005 to August 
3E‘2006." 

Another disadvantage with using the last 12 months from the time of the interview as a reference 
period is the reduced comparability with other statistics that are based on a calendar year. Besides, the 
period of the last 12 months before the interviews varies between respondents depending on when the 
interview is conducted, so the reference period for the total sample would in fact cover a set of different 
12-month periods. 

The use of the calendar year will become a disadvantage, however, if data collection takes a long 
time. If the time between the interview and the calendar year becomes too long, recall problems may 
increase and thus reduce data quality. 

Although there is no “objective” clue for fixing a time-line between both options, concrete 
suggestions may be helpful. From the author’s point of view, a survey conducted in the first quarter of the 
following year should not cause serious recall problems in a calendar-year-approach while interviews 
conducted more than 6 months after the calendar year are likely to do so. 

Therefore, the following pragmatic approach is advised here: A calendar-year approach is 
recommended whenever it is possible to conduct international surveys in the first quarter of the following 



For long courses and educational programs, the school year remains a viable reference period. 

Ease of recall may also vary across recall periods with regard to the type of education considered. For example, 
it might be easier for university students to recall credit hours earned in an academic year than in a calendar 
year. 
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year. Only if data collection takes a long time and there is no possibility of speeding up the data collection 
process is a period of the last 12 months before the interview considered preferable. 

Should a longer reference period than one year also be used? 

Some countries also use a longer reference period than the last 12 months, in addition to using a 12- 
month period (e.g. Canada, Finland, Germany and the United Kingdom). As mentioned before, when using 
longer reference periods recall problems tend to increase. Therefore, it may be asked: Why should longer 
reference periods than one year be used in AL surveys if validity of results seems to decrease with 
increasing length of time periods? 

In interviews, some survey experts described to the author their tacit knowledge of a “telescope 
effect”, which can be described as follows: Respondents who participated in learning activities that were 
important to them and took place outside a one -year reference period tend to report these activities within 
the 12 months period rather than not reporting them at all. Therefore, a reference period of one year 
without a longer one in addition increases the risk of over-reporting learning activities in the one -year 
period. This may seriously affect participation rates. 

To avoid the “telescope effect”, the author proposes AL surveys to follow the Finnish example 
described in the question and answer section of chapter 5.1. Both the last 12 months and a longer reference 
period ought to be included in an AL survey in order to give the respondents the opportunity to report 
learning activities that were important to them. 

A longer reference period than one year also will give information in order to evaluate the connection 
of the respondent to the learning system as was done in the Canadian Adult Education and Training Survey 
(AETS) 2003 (see Statistics Canada 2004, p. 22ff). It should be stated clearly, however, that another main 
purpose in adding an additional category for a longer reference period is to increase the validity of results 
for the one -year period. 

Conclusions 

1) AL surveys should cover an annual reference period for measuring participation in formal and non- 
formal education. Unless the data collection process takes too long, we prefer a calendar year over the 
last 12 months before the interview, as the calendar year more adequately meets the needs of statistical 
users and the need for comparability with other annual data sources. 

2) A longer reference period should be covered in addition, as this will increase the validity of results for 
the one-year period too. 

3) To measure participation in informal learning it would be useful to combine data for a calendar year 
and a shorter reference period, e.g. the last four weeks. 



This recommendation does not strictly accord with the AES Taskforce that recommended a period of the last 12 
months from the time of the interview. 

Results from the AES-survey in UK support this assumption, see Larsson 2006, S. 4. 
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► AL surveys should use an annual reference period for measuring learning activities and 
try to add a shorter reference period for Informal learning (e.g. four weeks). 

► Statistical users prefer a calendar year to a varying 12-months period. 

Only if time between data collection and the calendar year gets too long (6 months 
or more) a varying 12-months period will be preferable. 

► A longer reference period than 1 year should be added. Otherwise some respondents will 
report activities that happened longer than one year ago ("telescope effect"). 



Helmut Kuwan 2008, OECD AL Module 



3.3 Target population 

When defining the target population of an AL survey three questions need to he answered: 

• Should persons living in institutions he included? 

• Should foreigners / migrant workers he included? 

• Which age limits should he recommended? 

To include persons living in institutions (e.g. residential homes for the elderly, nursing homes, 
prisons, etc.) in an AL survey would he a very hig complication in designing the sample and in conducting 
the survey, and would increase survey costs significantly. Since only a very small part of the population 
lives in institutions, we recommend that this group should not he included in an AL survey. Only people 
living in private households should he part of the target population. 

Should an AL survey include migrant workers and non-national residents! In most countries, these 
are not just a “residual group”. However, national policies among OLCD members on immigration and 
non-national workers vary greatly as do the legal definitions (landed immigrants, permanent residents, 
refugees, temporary workers etc.) Countries such as Canada, the US and Australia have a large number of 
legal immigrants who are permanent residents and have yet to become citizens. There are also non-national 
workers on temporary employment visas. These groups or at least the family members selected for 
immigration tend to be highly educated, with a capacity in the official language. 
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In some other OECD-countries, it would only be possible to cover migrant and non-national workers 
in a fully representative approach with considerable effort due to possible language problems.'^ While 
incurring significantly higher costs, these activities in most countries still would not result in a sufficiently 
high number of interviews for quantitative analysis based on nationality groups. 

The final decision is a matter of national policy priorities. Some countries (e.g. Canada) would be 
extremely reluctant to exclude groups such as immigrants from the sample for policy reasons as well as the 
fact there are large numbers of non-national residents comprising a significant proportion of the population 
in certain areas. 

One feasible possibility is to include foreign residents or migrants without changing the survey 
instruments.^^ In this approach, only non-nationals whose command of the resident country's language is 
adequate to conduct an interview are surveyed. The assumption seems justified that this group has been 
integrated more successfully than others. While in some countries the survey would reach only a limited 
group, at least some information could be collected in doing so.'’ 

The question of lower and upper age limits is controversial. In the EU AES, the final recommendation 
was to focus on a target population from 25 to 64 years. Eor countries with different priorities, it is possible 
to widen the target population in national surveys and quite a few countries do so (see Larsson 2006, p. 3). 
Since the target population is a key issue for an AL survey, this question will be discussed in some detail. 

The focus on the population of 25 to 64 years chosen by the EU AES has advantages and 
disadvantages. The main advantage is that this pragmatic specification helps reduce some rather 
complicated problems. This age limit probably leads to a rather small number of persons who are in initial 
education. Because of this, the final EU AES questionnaire does not have to deal with the problem of 
separating initial from continuing education. Some countries, however, have included such questions in 
their national questionnaires. 

On the other hand, excluding those less than 25 years of age can have serious consequences. A 
considerable number of 18-24 year olds are not in initial education and it is very likely that this group often 
will participate in adult learning activities - sometimes the same activities as those 25 years and over. To 
maintain an age limit of 25 years or more will prevent a comprehensive examination of adult learning 
activities. 

In addition to this, 18-24 year olds who are not in formal education are likely to have lower levels of 
education. Among them are young people who dropped out of high school and those who did not pursue 
formal education beyond high school. Information about further non-formal or informal learning 
opportunities of this group is very important. 

If possible, it is recommended here to maintain a minimum age of 18 years (younger people do not 
belong to the adult population) while trying to exclude initial education. Excluding specified age groups 



To include foreigners whose knowledge of the language is not adequate, the following may be necessary: 
translating questionnaires into various languages, re -translating the answers, involving an interpreter in the 
interview, etc.. 

Germany chose this option in 1997 while in previous national AL surveys only Germans belonged to the target 
population. 

When following this approach it will be useful to include in data sets and reports information on the number of 
interviews that could not be completed because of language barriers. 

Thanks to Patrice de Broucker from Statistics Canada for stating this point. 
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that are attending high school and university will help to do so, hut will not he enough. Some detailed 
questions will he necessary to identify initial education. 

Should there he an upper age limit? The consequences of setting an upper age limit such as excluding 
those age 65 years of age and over do not appear as serious as the consequences of decisions on minimum 
ages discussed earlier. Although it varies hy country, seniors have low rates of participation in adult 
learning. However, with an aging population in many OECD countries, the participation of seniors in 
learning activities will become of greater policy interest, given the growing recognition of the benefits that 
all adults, and not just those who are economically active, can gain from engaging in learning.^® 

In September 2006, Germany started a representative study dealing with adult education among elder 
people from 45 to 80 years of age. This study, which is connected to the national AL survey, illustrates 
some of the problems of high upper age limits. Questions in which the situation of the potential work force 
is a major issue do not make sense for retired people. Training obtained by older persons is not as likely to 
be job-related but rather will reflect personal interest. As a consequence, an additional set of questions 
referring to the heterogeneous situation of older people has to be developed dealing particularly with issues 
like health and social participation as well as specific barriers and motivation. 

A very high, or no, upper age limit raises other questions too. One issue is how to treat persons in 
institutions or persons with serious health problems. Another question concerns the readiness of very old 
people to participate in an AL survey. 

Conclusions 

1) Only people living in private households should be part of the target population of an AL survey. 

2) Resident foreigners and migrants should be included in an AL survey, if this will be possible without 
changing the survey instruments. Although the interviewed group will be integrated more successfully 
than other groups, this approach is preferable to the exclusion of resident foreigners and migrants. 

3) Age limits will vary according to each country’s political preferences. It is recommended here that the 
lower age limit be 1 8 years. There should be an upper age limit, too. Lor the reasons described above, 
the author’s personal recommendation for an international AL survey would be a target population 
from 18 to 64 years. 

4) A pragmatic option would be to focus on the population of 25 to 64 years as suggested in the EU AES 
in order to reduce some problems like separating initial from continuing education. 

3.4 Sample design: Address Random, Random Route and Quota design 

Surveys are based on different sampling-designs. Does the choice of sample design affect the survey 
results in a major way? 

The following section mainly refers to different sample designs of f2f surveys. In an Address-Random 
sample (AR), all persons belonging to the target group have the same chance to be part of the sample. The 
target person is fixed by a given address from a register. If the register data are up-to-date and include all 



In Germany, for example, more than 50% of young people leaving the “dual system”, whieh is the most 
frequently visited initial education there, are older than 18 years. 

In addition to that, it will be important to keep an eye on the development of the average retirement age in 
OECD countries. 



18 




EDU/WKP(2008)8 



relevant groups (e.g. some registers have problems with covering migrants) this sampling principle will be 
the first choice from a methodological point of view. However, in some countries (e. g. Germany) where 
registers are organised locally this is a very expensive sampling method which is hardly ever applied, as 
many local administrators need support in drawing a sample and ask for rather high fees if addresses are 
used in surveys. 

If AR samples are too expensive or register data are not up-to-date usually Random-Route samples 
(RR) will be applied. In RR-samples, all households with persons belonging to the target group have the 
same chance to be part of the sample. The interviewer first selects the household on a fixed random walk 
within a fixed sample -point and in a second step uses a systematic random key for selecting household 
members. 

Contrasting with these random-sampling principles are quota samples, where the interviewer is free to 
choose any target person suiting the given quota criteria. Even if these criteria represent the population 
structure exactly, other variables may differ considerably, as some methodological studies have shown. 
Therefore, an AL Survey should not be based on a quota sample. 

Recently, more and more pre-selected samples of persons willing to participate in interviews have 
come up in surveys. Even if target persons are drawn at random on the basis of these pre-selected 
addresses, this is not strictly random sampling because the pre-selection prevents persons who belong to 
the target population from having the chance to be part of the sample. 

Therefore the sampling principles we recommend for an international AL survey are Address - 
Random and Random-Route Samples for f2f surveys or Random Digit Dialling (RDD) for telephone 
surveys. RDD usually starts from existing telephone numbers and varies the last digits at random. This 
method avoids the regional clusters of a RR sample. However, problems may arise if cellular phones 
become more and more the only telephone extension of a household (see 3.1). As mentioned before, this 
problem could be solved if valid samples of cellular phones were available. 

As samples based on register data are not feasible in every country it will not be possible to harmonise 
the sampling principles in an international AL survey completely in f2f interviews, because participating 
countries will use both AR and RR samples. Therefore, it is very important to check if differing results are 
to be expected because of these different sampling principles. 

In an AR survey each individual has the same probability of becoming part of a sample. In contrast, in 
RR surveys the probability of individuals becoming sample members varies with the size of the household 
in step 2 of the selection. Therefore, AR surveys lead to representative data for the population 
(individuals) whereas a direct use of RR results leads to representative data for households?^ 



One example may illustrate this problem. If the variables “age” and “sex” constitute the quota matrix, the 
sample usually will represent the population structure quite well with regards to these two variables while 
other variables like “educational level” or “labour force participation” may differ considerably from the 
structure of the population. 

One example may illustrate this: Person A lives in a one-person household selected in step 1 of a RR survey. 
Her probability of being selected in step 2 is 100%. Person B lives in a household of 10 target persons. If 
this household is selected in step 1 of a RR-survey, the probability of person B to be selected in step 2 is 
only 10%. RDD samples face the same problem here. 
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A RR survey can be transferred from a household survey to a survey for individuals by a design 
adjustment, which corrects unequal selection chances varying with household size. After this design 
adjustment, there should be no difference between the samples theoretically. 

Design adjustment is a minor effort. It should be very clear, however, that comparing the results of 
AR and RR surveys without design adjustment will lead to a comparison of population structures 
(individuals) with household structures, which would be a mistake. 

Conclusions 

1) An international AL Survey should be based on a random sample (Address-Random, Random-Route 
or RDD in telephone surveys). Methodologically, AR (selection from registers) is considered best 
practice in f2f-interviews. 

2) As national sampling frames vary, the choice between AR and RR should be left to the countries. 

3) Results from AR and RR surveys are comparable only after design adjustment. Without design 
adjustment, individuals (population structures) will be compared with households. 

4) Quota sample designs and samples based on a pre-selected population willing to participate in 
interviews should be avoided. 



Figure 6: 
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► Resident foreign nationals and migrant workers can be feasibly included by using the 
same survey instruments. Some information is better than none! 

► Address Random is best practice in f2f-surveys, but not always feasible. 

RR surveys are comparable only after design adjustment. 

► Age limits will vary according to country's preferences. The author's preference 
is 18-64 years. 



Helmut Kuwan 2008, OECD AL Module 



Analysing the results of a family survey in Germany showed, however, that the sample-realisation in the AR 
sample was better. There was a very controversial discussion whether this result represents a general 
pattern or not, see Alt, C., Bien, W., Krebs, D. 1991 (available in German only). 
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5.5 Random selection of learning activities 

A remarkable number of persons take part in multiple learning activities. Average numbers vary not 
only between countries^"^ but also between formal and non-formal education and informal learning. In non- 
formal education and informal learning, more learning activities are expected than in formal education. 
Only very few multiple learning activities will happen in formal education. 

There is a trade-off between obtaining a maximum of information and the limits caused by response 
burden. Collecting detailed information on all learning activities in the reference period is not a feasible 
approach, as this would take too much interview time. Therefore, recommendations on this issue have to 
deal with two questions: 

How should learning activities be selected? How many learning activities should be selected? 

Statistically, a random selection of learning activities as first put into practice by Switzerland, Finland, 
Sweden and the USA is preferable but this option also increases the response burden since in a first step all 
learning activities during the reference period have to be listed. Therefore, it is discussed first if simpler 
approaches with a smaller response burden might be acceptable too. 

A relatively small response burden would result from choosing one learning activity in a direct 
approach, which avoids collecting information on all learning activities. Examples of this direct approach 
are to choose the most recent activity or identify the ‘main’ training course attended — the training course 
on which most time was spent. 

The simple non-random method of selecting the most recent activity may lead to an acceptable 
approximation for some aspects but will cause a bias for other important aspects. In the German national 
GET report system (BSW), data based on the most recent activity of non-formal learning covered aspects 
like providers, fields, etc. quite well whereas volume results were biased because longer activities are more 
likely to be selected. When selecting the main activity instead, a bias on volume is systematically 
inherited.^® 

Selecting one activity at random reduces this bias and increases the response burden since all learning 
activities during the reference period have to be listed first. Still, this selection is not unbiased. 
Respondents who had attended one course only might influence the results more than other participants 
because this selection covers learning activities of respondents who had attended only one course at a 
100%-rate while it includes only a sample of learning activities of other participants. 

Recommendations have to take into account not only the varying number of learning activities in the 
fields of formal and non-formal education and informal learning - and policy interest - but also the 
interview time in a national AL Survey, which may vary. Presently, there seems to be more policy interest 
in non-formal education than in informal learning. Recommendations should not ignore this fact. 

As stated before, only few multiple learning activities will happen in formal education. One feasible 
option might be to focus on two activities or the most important activity here. Covering more than two 
learning activities in this area probably will not lead to much better information. 



In the Canadian AETS 67% of training participants in 1998 reported only one event, whereas Australia reports 
in 1997 a number of 43 %. 

Another problem with using the most recent activity is related to seasonality effects. 
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In non-formal education the situation is different. A variety of learning activities may exist. As a basis 
for random selection, a list of learning activities has to be created first. In a second step, activities are 
selected at random. According to some recent AL survey experiences the sampling procedure itself does 
not seem to cause too much difficulty. However, there are problems with very high numbers of activities, 
as experience from the UK illustrates: 

“A few people had done around 20 learning activities and were finding the section about listing non- 
formal activities tedious. UK provided a note to interviewers telling them that recording details of the 6 
most recent activities was sufficient, as long as they recorded the total number of activities.” (Larsson 
2006, p. 33). 

Results from Finland and Sweden in the field of non-formal education have shown that if less than 
three learning activities are selected the estimates for some aspects (e.g. fields of learning and learning for 
personal reasons) will be more uncertain.^^ Nevertheless, the objection based on the UK experience seems 
to be justified. 

Considering all of the above, our preferred option is a random selection of three activities in non- 
formal education. Taking up the suggestion from UK experience, recording details of six most recent 
activities should be sufficient, as long as the total number of activities is also recorded. 

For informal learning, random selection is not a feasible option as the number of learning activities in 
this heterogeneous field can be very high. Selection could take place by focussing on the most recent 
activity or the most important activity for the respondent (see 5.3.). While selecting the most recent activity 
reduces recall problems there seems to be bigger policy interest in selecting the most important activity. As 
both options have their specific advantages no recommendation is given here. 

It is very important, however, to use the same criterion for selecting an activity in each country. Using 
different criteria, (e.g. the most recent vs. the most important or longest activity) would lead to different 
forms of bias and seriously affect levels of international comparability. 

In countries where the response burden for a random selection is considered too high, e.g., when very 
short questionnaires are used, one learning activity in each field (formal and non-formal education and 
informal learning) could be selected in a non-random way. As mentioned before, it will be very important 
in this case to use the same criterion for selecting an activity in each country. 

Conclusions 

1) In formal education, where few multiple learning activities are to be expected within a 12 months 
period, it is recommended to select the most important activity instead of a random selection. 

2) In non-formal education, information should be collected on three randomly selected learning 
activities. A limit for listing detailed activities, e.g. six activities, should be accepted. 

3) Informal learning seems to be too heterogeneous for random selection. Other criteria will have to be 
used, e.g. selecting the learning activity considered most important by the respondent. 



Larsson 2006, p. 5. 

Results from Germany support the suggestion from UK. In Germany, less than 0,5 % of the population 
participated in more than 6 activities of non-formal education in one year. 
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4) In countries where very short questionnaires are used and little interview time is available, one 
learning aetivity in eaeh field might also he seleeted in a non-random way. In this case, all eountries 
should use the same criterion for selection since different forms of bias would result from the use of 
different selection criteria, and this would affeet international comparability seriously. 



Figure?: 

Random selection of learning activities: 

A trade-off between maximum of information and limits caused by response burden 



Do 

★ Random selection of 

learning activities if it is feasible 






Do not 

★ Make the response burden 
too heavy 




★ If it is not feasible: Use the same 

selection criteria in different countries 



★ List all activities of informal 
learning 



► Random selection of more than one learning activity (e.g. 3) is very useful for 
non-formal learning. A limit for listing activities (e.g. 6) should be accepted. 

► In formal education only few multiple activities in one year are to be expected. 
Therefore, selecting the most important activity seems to be acceptable. 

► As informal learning is very heterogeneous, selection criteria like the most recent or the 
most important activities may be helpful. Different countries should use the same 
selection criterion. 



Helmut Kuwan 2008, OECD AL Module 



3.6 Carrier surveys 

While in 2002 the question of how to integrate an OECD AL module into a carrier survey was a main 
issue, the EU AES survey has changed this situation considerably. However, some aspects of this earlier 
discussion remain important. 

Should an OECD AL module be part of a survey dealing only with AL -topics or should it be part of a 
more general survey dealing with several topics? Methodologically, a separate AL survey is the preferable 
solution. 

A separate AL survey makes it possible to cover a wider range of content including more background 
variables that may influence participating in learning activities. However, it would be much more 
expensive than a multi-topic survey. Decisions will vary between countries according to policy interest and 
available budgets. Nevertheless, we consider a separate AL survey to be best practice because of the 
methodological advantages and the higher analytical potential of the survey data collected. 

The OECD AL module could be part of a newly developed or an already existing survey. In the EU 
AES, both options exist as well. Integration into an existing survey could include national surveys or the 
Labour Eorce Survey (EES). With the EES, two problems exist: the EES usually allows only a very limited 
set of AL questions, and in some countries it allows the use of proxy interviews, which cannot be 
accepted here (see 3.7). 

Which other international surveys could be considered carrier surveys for an OECD AL module? 
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In surveys like the Adult Literacy and Lifeskills Survey (ALL) or the International Adult Literacy 
Survey (lALS) the response burden already is very high. An additional AL module would make it 
unhearahle. As for Programme for the International Assessment of Adult Competencies (PIAAC), it is too 
early for a recommendation since key elements in the PIAAC design are yet to he defined. According to 
first impressions, the situation here might turn out to he similar to ALL hut it is important to watch the 
development of PIAAC closely in order to identify possible connections early on. 



A more promising alternative might be the International Social Survey Programme (ISSP) carried out 
in more than 30 countries where a drop-off questionnaire is already part of the study concept. Using a 
drop-off questionnaire would also somewhat reduce the response burden in other surveys. 

The optimal option, however, would be to integrate the OLCD AL module into the LU ALS and 
existing national surveys in non-Luropean countries. 

Conclusions 

1) A separate AL survey is better practice than a multi -topic survey with an AL module, as a separate 
AL survey supports a wider range of content. 

2) With regard to the OLCD AL module, integration into the LU ALS and national AL surveys in non- 
Luropean countries would be first preference. 

3) Another option might be to integrate the OLCD AL module as a drop-off-questionnaire in an 
international survey. 

3 . 7 Proxy interviews 

Should proxy interviews be allowed in adult education surveys? 

In proxy interviews, one member of a household provides information on another household member. 
As some countries allow proxy response in their national labour force surveys, supplements to these labour 
force or general population surveys often inherit, so to speak, the use of proxy responses. 

Although this is a very basic question, not much systematic information is available on the effects of 
proxy interviewing on AL data. Some methodological hints on this issue come from Norway and Canada 
and further information from Germany will be available soon.^^ A Norwegian study referring to the effect 
of proxy interviewing on employment shows that parents tend to underestimate the employment rate of 
students. In Canada, evaluation based on focus groups and interviews concluded that proxy responses 
were the source of several serious errors. Consequently, it is no longer used in the Canadian ALTS.^’ 



A drop-off-questionnaire is an additional questionnaire, which the interviewer hands out to the respondent after 
a f2f-interview. 

In Germany, a research project presently deals with the validity of proxy interviews on AL issues. Results will 
probably be available in 2008. 

The relative underestimation found was between 5 to 10 per cent of the published figures for students, Solheim, 
L., Harland I., Lagerstrom, B.; Proxy interview and measurement error of employment for the age group 
16-29: The Norwegian Labour Force Survey, Statistics Norway (no year). 



The assessment looked at different issues: how well respondents understood the questions, terms, and 
definitions such as program and courses; the ability of respondents to recall specific training events; and 
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Analyses of LFS-data also showed problems of inaccurate information due to proxy interviewing (see 
Larsson 2006, p.9). 

Taking into account this information, it seems very unlikely that one member of the household would 
be able to give valid information on all AL participation of other household members, particularly in the 
field of training within companies. It is even less realistic to expect valid results on details of an AL 
activity (e.g. the use of technology) and on subjective items such as whether or not the course met 
expectations. 

Conclusion 

No proxy interviews should be allowed in adult learning surveys. 



response burden. Summary Report: Adult Education and Training Survey Questionnaire Evaluation 
Thompson Lightstone May 92. 
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4. Discussion of basic concepts in the European Adult Education Survey 

4.1 The concept of formal and non-formal education and informal learning 

As mentioned before, quite a few countries in Europe have decided to use the EU AES concept of 
formal and non-formal education and informal learning in their national AL surveys and some use it 
already. Meanwhile this is the most widely applied concept in international AL surveys. Integration of this 
concept in national surveys is an important progress for international comparison in the AL field. 
Therefore, this chapter discusses the basic concept of the EU AES. 

It has to be stated that this section does not investigate in detail the broad variety of issues addressed 
in the EU AES concept. Partly this will be done in the following chapters dealing with specific issues. 
These chapters will refer in some detail to Eurostat’s “Classification of Learning Activities (CLA)’’ but not 
to the Single Learning Activities (SLA) also described there. According to the e-mail discussions with 
country experts SLA seems to be a rather abstract construct, not an operational concept that gives input for 
constructing a questionnaire. 

This section presents an overview of the basic conceptual elements in the EU AES, reflecting some 
experiences of surveys based on this concept and discussing some basic related issues. 

In order to develop concepts and methods for a harmonised EU survey on participation in education 
and learning by adults, the Task Eorce on Adult Education Survey (TE AES) was created. The main 
conclusions in the final report from this Task Eorce are presented in the following rather long quotation 
taken from the report prepared by Ann -Charlotte Larsson for the OECD AL Module (Larsson 2006, p. 2f.). 

Main objective of the AES 

“The main purpose of AES is to describe participation in education and learning by adults. Learning 
activities are defined as “any activities of an individual organised with intention to improve his/her 
knowledge, skills and competence” . Intentional learning is defined as “a deliberate search for knowledge, 
skills, competences or attitudes of lasting value”. The learning activities are divided into formal education, 
non-formal education and informal learning, which are defined according to the classification of learning 
activities. The criteria used for distinguishing between the three categories in the classification of learning 
activities are “intention to learn”, “institutionalised” and “included in the national framework of 
qualifications (NFQ)” (see the figure below). 



The final report from this task force was presented and endorsed by the group of Directors of Social Statistics in 
September 2004. 
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Figure 8: 

Concept of formal education, non-formal education and informal learning in the 

European AES 

Developed by Eurostat / Taskforce AES 




Helmut Kuwan 2008, OECD AL Module 



The purpose of the EU AES is to provide information about the adult population's participation in 
education, training and learning. Information is also produced about people's opinions and experiences of 
adult education and training, their willingness and need to participate in it, and the obstacles they face. In 
addition, adults' information technology skills will also be studied, along with knowledge of foreign 
languages and participation in cultural and social activities. Eor the first time, internationally comparable 
data will be produced on such a large area of adult education and training. 

One important purpose of the EU AES is to provide information about non-leamers, who are currently 
especially interesting for policy-makers. 

The following four groups are possible to identify with the help of the AES: 

• Non-learners 

• Learners only in formal and/or non-formal education 

• Learners combining participation in formal and/or non-formal education with informal learning 
activities 

• Learners only in informal learning activities 



Another purpose of the survey is to identify the adult learner. The adult learner is defined as a person 
who has left initial education. One definition of initial education that was discussed in the Task Eorce was 
that it comprises any education that the person participates in before a break of sufficient duration. It was 
decided that students in initial education should not be excluded from the survey. 
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To be able to identify tbe adult learner requires questions describing tbe path through the educational 
system and reasons for long breaks from learning. This kind of information can probably be collected with 
sufficient quality through information from registers at statistical offices in the Nordic countries. EU AES 
decided that it should be possible to distinguish between students in initial education and persons 
participating in continuing adult education through the questions in the EU AES. It was however judged 
that it would be difficult to construct harmonised questions on this matter which would give comparable 
results between countries.” (Larsson 2006, p. 2f.) 

Content issues in the EU AES 

The EU AES is structured in different modules which include the priority issues stated in section 3 
and some other issues such as participation in social and cultural activities, and use of ICT (information 
and communication technology) and ICT skills (for more details see Larsson 2006, p.8). The following 
section is an excerpt from Ann-Charlotte Larsson’ s report mentioned earlier. It gives an overview on how 
the general concept of the EU AES was put into practice by participating countries and on survey 
experience with regard to key conceptual issues (Larsson 2006, p. 9L): 

“... There are both variables and questions in the questionnaire. Eor the variables, the wording of the 
questions should be adapted to national conditions. This means that the wording of the questions will be 
different for different countries. The questions in the AES questionnaire should be directly translated into 
the national language. 

Eor each question/variable in the AES a level of preference for inclusion in the survey has been 
established. The questions/variables with level 0 are core questions/variables and should be included in the 
survey for all countries. If for any reason it isn’t possible to cover all questions/variables in the survey the 
questions/variables with the lowest level of preference (9) should be excluded first. . . 



Eormal, non-formal and informal learning are defined according to the classification of learning 
activities. The EU AES contains variables for identifying formal and non-formal education and informal 
learning (see the EU AES manual). This means that each country has adapted the variables to national 
conditions when constructing questions. Eor formal and non-formal education the definitions in the 
classification of learning activities are followed. The division into classes and sub-classes for non-formal 
education are however not entirely followed. Eor informal learning some of the sub-classes suggested in 
the classification of learning activities are excluded from the survey. The reason for this is mainly that it 
would have been difficult for the respondents to make the separation, for example between learning from 
colleagues, on the job training and learning through a learning group. 



There was also a discussion about the difficulties to separate informal learning from random learning 
for the respondents. It is stated in the classification of learning activities that informal learning should be a 
deliberate search for knowledge, skills etc of lasting value.” 

The difference between variables and questions mentioned here is important. As for questions, the 
wording is fixed. Questions and categories should be directly translated into national languages. Variables 
allow for varying wordings of the questions and answer categories often vary strongly according to 
national conditions. 

Although using the EU AES concept of measuring learning activities in surveys in different European 
countries is a very important step forward, some aspects need further discussion. Before presenting some 
experiences from the EU AES surveys already conducted, a more general issue will be discussed. 
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The fact that educational experts in different European countries welcomed the EU AES concept as a 
common basis for measuring learning activities does not indicate that it will he easy to communicate to 
persons who are not educational experts. Quite the contrary: This concept will prohahly he quite difficult to 
communicate in interdisciplinary discussions, and it will he even more difficult to communicate to 
respondents. For this reason, countries participating in the EU AES have tried “translating” this concept 
into learning activities and providing interviewers with special training. Although this seems to have been 
a feasible way, further effort in making the EU AES concept simpler and easier to communicate would be 
a very important step. 

Some experiences from the EU AL surveys already conducted 

So far, most countries that have implemented the EU AES have not reported major difficulties with 
classifying the learning activities into formal and non-formal education and informal learning although 
some double reporting occurred. Respondents had some problems identifying differences between certain 
categories (e. g. “seminars, workshops” and “courses”). Many of the problems reported refer to the 
separation of “guided on the job training” from other categories. Results also indicate that in general a 
clear separation between the work-related and personal contexts of activities would be helpful. This report 
presents some suggestions on this issue (see 5.3). 

The survey experiences show that it is particularly difficult for the respondents to distinguish between 
incidental learning and deliberate learning. In the survey, interviewers need to give considerable 
explanation. Therefore, training of interviewers is considered very important before conducting the survey. 

The EU AES surveys from 2005 to 2007 may be considered pilot surveys. Experiences during 
implementation will probably lead to some changes for the next EU AES surveys in 2010/2011, and in this 
process it should be possible to consider experiences from non-European countries as well. 



As expected, these analyses show that there is still some work to be done. In the following section one 
of the problems often reported in the EU AES, namely the separation of informal learning and random 
learning will be discussed. 

Conclusions 

1) The EU AES concept of formal and non-formal education and informal learning represents 
remarkable progress. 

2) However, this concept will be difficult to communicate in interdisciplinary discussions. In an AL 
survey, it requires “translation” into learning activities. 

3) Separating certain categories of non-formal education from informal learning seems to be one of the 
major difficulties. 
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Figure 9: 

Concept of formal and non-formal education and informal learning 



1 


Basic principle for the EU Adult Education Survey (AES) 

★ Important concept development for Education and Training 

★ Further discussion is needed on some aspects. 


2 


The concept is difficult to communicate to non-experts. 

★ Hardly applicable for direct questions in a population survey; 
"translation" into learning activities is needed. 

★ Difficult concept in interdisciplinary discussions 



Helmut Kuwait 2008, OECD AL Module 



4.2 Informal learning and random learning 

A remarkable part of adult learning activities takes place outside of the educational system. In a 
knowledge society, informal learning activities will increase further and so will policy interest in this issue. 
Qualitative research shows that even “learning en passant” may have big and lasting impact. 

In the EU AES concept described in section 4.1, informal learning is a “residual category” for 
activities which are considered as neither formal nor non-formal education. One of the most important 
differences between non-formal education and informal learning in the EU AES is that no professional 
teacher is involved in the informal learning activities. In this concept, the intention to learn marks the 
difference with “random learning”. 

The EU AES tries to exclude random learning. Eeedback on the EU AES shows that this is quite 
difficult from the point of view of the respondent. Starting from this difficulty, we would like to ask a 
somewhat heretical question: Is a strict exclusion of random learning necessary from a conceptual point of 
view? 

The main reason for excluding random learning is to avoid recorded learning activities coming close 
to a rate of 100%. In this case, there would hardly be any analytical value resulting from this figure. 

Instead of strictly excluding random learning, another option might be to include it partly and try to 
restrict the segment covered. This could be done by asking whether the random learning activity had a 
substantial effect. 

There seems to be some reason for including this segment. Eor instance, one could reasonably ask 
why learning achievements should be less important than learning efforts. Presently, activities that are 
started with an intention to learn, and where no learning effect was achieved are included in AL surveys 
like the EU AES, but activities where something substantial was learned without the intention to learn are 
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not included. An extension of the concept in the direction of random learning with substantial effects might 
have two advantages: 

• Problems in separating non-formal education and informal learning probably will decrease; 

• The concept would shift somewhat from input to outcome orientation. 

As random learning is not covered in most AL surveys, no tested instruments on this issue are 
available yet. The recommendation given here is to encourage testing in this direction. 

The following graph illustrates the suggested concept enlargement. Although the proposal 
complicates the graph, it might help in facilitating the interview situation. 



Figure 10: 

Extended concept of formal edncation, non-formal edncation and informal learning 

including random learning 

Extended concept (based on EU AES) 




Helmut Kuwan 2008, OECD AL Module 
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5. Participation in formal education, non-formal education and informal learning 

As each country conducting the EU AES will deliver quality reports to Eurostat, it would have been 
very useful for the OECD AL module report if this information had been available already. Since this is 
not possible, the following section only refers to the limited feedback on the EU AES, which is presently 
available. 

5.1 Participation in formal education and training 

Concepts and definitions 

The concept of formal education in the EU AES as explained in the CLA refers to the “educational 
ladder” of ISCED 97 where completing one level of education gives access to a higher level. The criterion 
that distinguishes formal and non-formal education is “whether the activity is designed to lead upon 
successful completion to a learning achievement that is possible to position within the National Eramework 
of Qualification (NFQ). “ (cit. from CLA -Manual, p. 18). Therefore, the NEQ is a key conceptual guide in 
the EU AES. 

NFQ covers education and training. It is defined as “the single, nationally and internationally accepted 
entity, through which all learning achievements may be measured and related to each other in a coherent 
way and which define the relationship between all education and training awards” (cit. from CLA-manual 
p. 19). 

One basic difficulty of international classifications in the field of formal education is that they refer 
(and have to refer) to general terms and definitions that will be more clear in some national contexts and 
less clear in others. As this problem reflects differences between educational systems, the correspondence 
of aggregate concepts to national system usually will vary. This is also true for the separation of initial 
education and continuing education.^^ 

Surveys are “moments of truth” following conceptual work. Surveys have to translate concepts and 
definitions into existing national educational systems and try to help the respondent understand their 
meaning. Therefore, it is very interesting to see how the results of this translation process worked in the 
AL surveys already conducted. 

In some non-European countries reviewed here, the focus in the surveys seems to be on non-formal 
education and informal learning rather than formal education. Australia asks some additional questions on 
formal education referring to the most recent activity, while surveys from the USA mainly seem to refer to 
non-formal education and informal learning.^'* Canadian surveys cover both formal and non-formal 
education. 

Questions and answer categories 

Because of varying educational systems, questions used for measuring formal education vary (and 
have to vary) considerably between countries. While some countries ask relatively short questions, others 
present rather long lists of formal education activities or present lists of providers of education to measure 



Separation of initial education and continuing education is an important issue which is not yet solved. Excluding 
19-24 year olds reduces this problem, but will lead to a loss of information. Therefore, we do not 
recommend excluding this age group (see section 3.3). 

For more detailed information on formal education, the USA relies on the Integrated Postsecondary Education 
Data System (IPEDS) and the National Postsecondary Student Aid Study (NPSAS). 
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formal education. These examples may show that it will not he possible to recommend identical question 
and answer categories for capturing formal education in different countries. 

How well has the EU AES concept of formal education and training worked in country surveys so 
far? As expected, differences between countries exist. In some countries, most respondent understood quite 
well what “designed to lead to a qualification” meant, while there were considerable difficulties in other 
countries. 

Most countries report that using lists and show cards was very helpful. Nearly all countries put great 
emphasis on training interviewers. Often interviewers had to explain the overall concept to the 
respondents. While interviewer support no doubt is an important issue, usually interviewers are not experts 
in a country’s educational system, even after special training courses. Therefore, we reco mm end not 
relying too much on the interviewer for clarification of the issues. Concrete listings of learning activities 
should be helpful. 

A few countries suggested that contact be made with the target persons before the interview by 
telephone or to send them a letter. While some active learners might prepare themselves for the interview, 
there will be a negative effect on non-learners. Detailed pre-announcements of survey contents will very 
likely reduce participation of non-leamers and increase sample bias. Therefore, announcements before the 
interview should be avoided in AL surveys. 

Methodologically, a list with clearly specified learning activities - as in the Einnish AES - seems to be 
very appropriate, although the response burden is rather high. However, the approach worked well in the 
previous AES. Using rather short questions that include pedagogical terms like “upper secondary level” 
probably will not make things easier for many respondents. 

As stated before, no recommendation for questions and answers can be given here. The Einnish 
approach combining questions on formal and non-formal education is documented not as a role model but 
as a methodologically attractive example that worked well in practice despite its rather high response 
burden. 

Example questions on formal education from Finland using an integrated concept (questions cover both 
formal and non-formal education) 

F lA Have you participated in training at the following general education institutes in the past 
12 months or earlier? 



Answer alternatives in question: 



Never studied (...) 

In the past 12 months (. . .) 

Or only earlier? (. . .) 

Don’t know (...) 
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01) Primary school or lower secondary school? 

02) Comprehensive school or middle school? 

03) Upper secondary school? 

04) Evening school (adult or evening upper secondary school or middle school)? 



F IB Some forms of vocational training are listed below. Have you attended any in the past 12 
months or earlier? 

05) Apprenticeship training? (Periods of courses taken at vocational adult education centres or vocational 
education institutions.) 

06) An employment-promoting course paid for hy labour authorities (labour policy training)? 

07) Training for youths at vocational or professional education institutions? 

08) Training arranged specially for adults at a vocational education institution or a vocational adult 
education centre in order to get a vocational or professional qualification (polytechnic education and 
private students included)? 

09) . . . for other purposes? 

10) Military institute (not conscript service)? 



F 1C The following is a list of institutes that offer both vocational and general interest courses 
and studies. Have you studied at: 

1 1) A folk high school or folk academy in order to get a vocational or professional qualification? 

12) A folk high school or folk academy for other purposes? 

13) A conservatory in order to get a vocational or professional qualification or in a music school or college 
(not in the adult application)? 

14) The music-school level adult department (former folk conservatory department) at a music school or 
college? 

15) A college of physical education (sports institute) in order to get a professional or vocational 
qualification? 

16) A sports institute in other training? 
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F ID I will list different study modes of universities and summer universities. Have you studied: 

17) For a basic degree at university? 

18) For a post-graduate degree at university? 

19) At university or its continuing education centre in a further training course? 

20) At university or its continuing education centre for a separate degree or Open University courses? 

21) At summer university? 

F IE In the following I will ask about the providers of training that take adults into account 
separately in the courses that they cater for. 

Have you in the past 12 months or earlier attended: 

22) An adult education centre? 

23) A language school or institute in Finland? 

24) A correspondence school? 

25) Training provided by an organisation, union or association? 

26) Courses or a study circle arranged by a study circle centre? 

27) A separate typing or word-processing school? 

28) A dance school or institute (modem or jazz dance or ballet)? 

29) A driving school? 

30) Studied by regularly following an educational programme series on the radio or television? 

F IF The following is a list of common modes of organising education and training. Have you 
participated in: 

31) Training provided by your employer at your workplace, in the employer's training facilities or in other 
training units (also applies to secondary jobs)? 

32) Training provided by a separate training enterprise or training centre (business-based training centres, 
consultants, etc.)? 

33) Guided on the job training? 

34) Conferences, seminars or other similar training? 
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F IG Have you participated in any other training? 

(...) No 

(. . .) In the past 12 months 
(...) Only earlier 
(. . .) Don’t know 



Figure 11: 

Important issues for measuring formal education in international AL surveys 



D 


National framework of qualiHcations 

★ In some countries this concept is easily understood. 

★ In other countries, respondents find it quite difficult to grasp. 


2 


Separation of initial and continuing education 

★ Separation of initial and continuing education is an important issue which is not yet 
solved. 

★ Excluding 19-24 year olds reduces this problem, but will lead to a loss of information. 


3 


Country questionnaires often differ considerably. 

★ Some countries ask lists of learning activities. 

★ Other countries use lists of providers, etc. 


■ 


Do not rely too much on the interviewer to solve conceptual issues! 



Helmut Kuwait 2008, OECD AL Module ■» 



5.2 Participation in non-formal education 

The concept of non-formal education in CLA refers to the definition of formal education. Non-formal 
education is defined as “any organised and sustained educational activities that do not correspond exactly 
to the above definition of formal education. Non-formal education may therefore take place both within 
and outside educational institutions, and cater to persons of all ages. Depending on country contexts, it may 
cover educational programmes to impart adult literacy, basic education for out of school children, life- 
skills, work-skills, and general culture. Non formal education programmes do not necessarily follow the 
“ladder” system, and may have a differing duration.” (CLA, p. 16). 

Primary statistical units are “institutionalised learning arrangements (organising frames) to learn one 
or more subjects.” (CLA, p. 24) 

Non-formal education is a very heterogeneous area. Not surprisingly, measurement concepts vary. 
Usually, empirical concepts try to combine measuring participation and purpose of non-formal education. 
Different approaches are: 

• The “two-step-approach” first asks about participation in different forms of non-formal education 
and second asks about the purpose of the learning (e.g. LU ALS); 
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• The “two-sector-approach” first asks separately about work-related non-formal education and 
then about non-formal education based on personal interest (e.g. BSW); the questions are based 
on detailed activity lists, particularly in the field of non-formal education based on personal 
interest; 

• Combined concepts ask about purpose (work-related) and participation in one question without 
using detailed activity lists in this first question (e.g. Canada, the USA, Australia). 

In examining effects of different instruments, no recent experience from non-European countries was 
found. Therefore, the following analyses refer to: 

• Experience from implementing the EU AES and; 

• Results from the BSW AES pilot study and Pretest comparing two different concepts. 



Quite a few problems in adapting the EU AES concept result from separating non-formal education 
and informal learning (see 5.3 for more detail), particularly with regard to on-the-job training. 

Another important issue is to avoid double counting. Statistics Sweden therefore reminded the 
respondent in each question that he or she only should report learning activities in addition to those already 
reported (see “question and answers”). 

Experience from Einland indicates that an additional longer reference period than 12 months is helpful 
to avoid the “telescope effect” in which activities which took place outside the reference period are 
reported within the reference period (see 3.2). 

The comparison of BSW and the AES refers to results from a pilot study in 2005 and a Pretest in 
Eebmary 2007. While in the pilot study (n= 715) the AES-concept lead to somewhat higher participation 
rates in formal education it lead to significantly lower participation rates for learning based on personal 
interest (14% vs. 29%, see von Rosenbladt/Bilger/Post 2005). This result corresponds with earlier findings 
about the importance of activity lists: 

“Participation rates vary depending on the type of question used. The simplest form is the unaided, 
direct question, e.g. ‘Have you attended any continuing education activities during the past 12 months?’ 
The pre-defined reply categories are yes/no. However, it is also possible to enquire about participation 
using aided questions, which directly refer to specific continuing education activities. 

A comparison of these two types of questions reveals that in unaided questions most subjects interpret 
the term "continuing education" as continuing vocational education. Therefore, the unaided question 
method is less suitable for surveying general continuing education. 

In direct questions, the subject must decide what he or she considers to be continuing education. 
However, the precise definition of continuing education is controversial even among experts. It is thus 
even more uncertain how the subjects define it. Therefore, aided questions are to be preferred for content- 
related and methodological reasons.” (Kuwan 2000, p. 12). 

Therefore, BSW used 17 categories to identify learning based on personal interest. 



The U.S. National Center for Education Statistics and Statistics Canada do the same. 
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As the BSW AES Pretest gathered information on more than one activity of non-formal education, it 
is possible not only to compare participation rates hut also the share of activities allotted to the non- 
vocational and to the personal interest sector. The differences are striking. According to the BSW concept, 
about 50% of all non-formal education activities are vocational while the AES concept accounts for more 
than 80%.^® 

Which figure represents the adult learning situation in Germany better? Provider statistics indicate 
clearly that this is the BSW concept, even though the indicators used in the data sources are not strictly the 
same (participants and participation cases). In Germany, the two-step-approach seriously underestimates 
participation based on personal interest. 

These results show that at least for Germany from a strictly methodological point of view the two- 
sector-approach is “better”. However, the use of the AES concept significantly improves international 
comparison in the AL field, and “platform effects” are not necessarily a major problem in international 
comparison if they do not vary between countries. Therefore, with regard to the criterion “international 
comparison” the EU AES concept will presently be considered “better”. As these remarks illustrate, what 
“better” means in this context depends on the weighting of these two criteria. Therefore, no 
recommendation with regard to the two-step-approach or the two-sector-approach is given here. 

Although in Germany the two-step-approach underestimates learning based on personal interest it is 
not known whether this will be true for other countries as well. Nevertheless, these apparent differences 
between the “two-step approach” and the “two-sector approach” call for further research on this important 
issue. If results in other countries confirm the German Pretest, the need for further discussion will increase. 

The “question and answer” section presents the Swedish questions which cover the EU AES concept 
very well. The German BSW questions are not listed as this concept was reconstructed in 2006 in order to 
develop an integrated concept with the EU AES. 

Questions and answer categories 

Sweden, EU AES: 



Examples of such education are training courses in the workplace, driving lessons, parenting 
course, language or computer courses, study circles in crafts or music. 



34a In the last 12 months have you participated in courses CH YES Proceed to question 34h 

or study circles in the workplace or in your free tinted , , 

□ NO ^ Proceed to question 35a 

□ DON’T KNOW ^ Proceed to 
question 35a 



34h 



Please list all courses and study circles that you 
participated in, including those which are 
ongoing. 



have Open question 
still 

If more than 6 - report the 6 most recent 
activities. 



IN THE LAST 12 MONTHS. 



Despite of the relatively small sample size, these results are remarkable. 
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35a 



In the last 12 months, have you taken private lessons? 

E.G. PIANO LESSONS 

N.B. IN ADDITION TO THE EDUCATIONAL COURSES 
ALREADY REPORTED BY THE RESPONDENT 



Q YES Proceed to question 35h 

Q NO Proceed to question 36a 

□ DON’T KNOW ^ Proceed to 
question 36a 



35h Please list all types of private lessons 
including those which are still ongoing. 

IN THE LAST 12 MONTHS 



received, Open question 



If more than 6 - report the 6 most recent 
activities. 



36a 



In the last 12 months have you participated in 
seminars, lectures, workshops or similar in the 
workplace or in your free time? 



Q YES -> Proceed to question 36h 
O NO -> Proceed to question 37 



N.B. THE ACTIVITY SHOULD INCLUDE SOME 
EORM OE EDUCATION. 



□ DON’T KNOW 

question 37 



Proceed to 



N.B. IN ADDITION TO THE EDUCATIONAL COURSES 
ALREADY REPORTED BY THE RESPONDENT 



36h 



Please list all seminars, lectures etc. that you have 
participated in. 

IN THE LAST 12 MONTHS 



Open question 

If more than 6 - report the 6 most recent 
activities. 



37a 



In the last 12 months have you received education, 
instruction or training directly in the workplace or in 
the work situation with the aid of advisers or other 
instructors? 

e.g. to operate a new machine or to learn new software 



O YES Proceed to question 37h 
O NO -> Proceed to question 38 

□ UNEMPLOYED ^ Proceed to 
question 38 



DO NOT INCLUDE HERE CASES WHERE THE □ DON’T KNOW Proceed to 

RESPONDENT ASKED A COLLEAGUE’S ADVICE question 38 
TO SOLVE A WORK TASK. 



N.B. IN ADDITION TO THE EDUCATION ALREADY 
REPORTED BY THE RESPONDEN. T 



37h 



List all the education, instruction or training which 
you received directly at the workplace or in the work 
situation. 



Open question 

If more than 6 - report the 6 most recent 
activities. 



IN THE LAST 12 MONTHS 
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Figure 12: 

Participation in non-formal education and purpose of participation 



1 


Concept A: Two-step approach 

★ Ask for participation in a first step 

★ Ask for purpose (job-related or personal interest) in a second step 


2 


Concept B: Two-sector approach 

★ Ask questions on job-related participation 

★ Ask separately questions on participation due to personal interest 


3 


Different results of Concept A and B? 

★ Concept A seems to cover more job-related activities than concept B. 

★ Concept A seems to cover significantly less personal interest activities than concept B. 


► Concept A seems to underestimate participation based on personal interest seriously, but 
further research is needed. 

► Recommendation: International AL surveys should be based on the same concept in 
participating countries. 



Helmut Kuwait 2008, OECD AL Module 



5.5 Participation in informal learning 

Concepts and definitions 

Despite the obvious diffieulties in eovering informal learning empirically it is very important to 
consider this form of adult learning in an AL survey as well, because excluding it might lead to false 
conclusions in trend analyses. If only non-formal education were considered, any potential shift to informal 
learning might go unnoticed and be interpreted as a decline in adult learning activities. 

Inclusion of informal learning in AL surveys widens the horizon of educational statistics and brings it 
closer to the “real-life-situation” of individual learners rather than focussing on learning in institutions 
only. This makes for important progress. However, the conceptual separation of informal learning from 
other learning activities is rather difficult. 

While Australia also used the LU ALS concept for informal learning, Canada, Germany, and the USA 
have developed different concepts. In Canada, the key concept is “self-directed learning”. 

Informal learning was one of the major problem areas of the LU ALS. Since feedback on the LU ALS 
concept of separating non-formal education and informal learning from countries that conducted the survey 
already is not very positive and all concepts mentioned above embody useful ideas, we have tried to design 
a new module on informal learning combining elements of different surveys. 

The approach presented here asks for information on a set of activities and it covers a broader variety 
of learning than the LU ALS. By the wording of questions and answers, it would equally avoid an 
“inflation” of informal learning activities. 

The LU AES does not really distinguish between job-related informal learning activities and those of 
personal interest. Since important aspects of the “learning situation” may differ considerably between these 
two fields, we consider this difference important and thus favour enlarging the AES-concept here. 
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Following the enlarged concept of informal learning outlined above, we suggest at the end of this 
chapter a module for this topic which is based on the questionnaires of Canada, Germany, the USA, the EU 
AES and some suggestions of the author. The basic structure of this module: 

• Draws a distinction between vocational informal learning and informal learning for personal 
interest. The questions for covering vocational informal learning borrow from the concepts from 
Canada, Germany, the USA and suggestions of the author while those for personal interest follow 
the EU AES; 

• Asks for the most important activity in the respective field; 

• Asks three questions (vocational informal learning) or two questions (personal interest) on details 
of the most important activity. 

One of the major problems in the EU AES was the separation of non-formal education and informal 
learning. Respondents sometimes reported learning from colleagues or supervisors at the workplace as 
informal learning. Our suggestion is to avoid most of the problems resulting from the overlap by adjusting 
the questionnaire better to the perspective of the respondents. This mainly refers to two categories: 

• We consider advice from colleagues or supervisors to be informal learning as this usually is a 
spontaneous reaction to a current problem and not a planned and structured learning activity; 

• on the other hand, we suggest to consider systematic and planned instructions from colleagues or 
supervisors (e.g. as part of company programmes on adjustment to a new job) to be non-formal 
learning. 

The module proposed below only covers some aspects of informal learning. Countries should be free 
to add further aspects according to national policy priorities. 

By adding a long list of additional activities, the overall participation rate in informal learning will 
increase. To avoid bias resulting from varying instruments, an international comparison of participation 
rates of informal learning should be based on the same activities. This recommendation may seem obvious, 
but it is not. Results based on lists that differed very much have been used in international comparisons. 
Different participation rates in informal learning, based on these analyses, reflected variation in the 
instruments rather than “real” differences between countries. 

Structure of the proposed new AL module for informal learning activities: 

A: Vocational informal learning 

VI: Activities in the last 12 months (6 categories) 

Additional option: in the last 4 weeks 

V2: Most important activity 
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Additional questions (referring only to the most important activity) 

V3: Volume (aggregate categories) 

V4: During work-time or not 

V5: Subject (field) of activity 

B: Informal learning of personal interest 

PI: Activities in the last 12 months (6 categories) 

Additional option: in the last 4 weeks 
P2: Most important activity 

Additional questions ( referring only to the most important activity) 

P3: Volume (aggregate categories) 

P4: Subject (field) of activity 
Questions and answer categories 

A: Vocational informal learning 

VI: Apart from the activities we have just discussed: Have you done any of the 

following activities to develop your job skills in the last 12 months? 

Additional option: in the last 4 weeks 

Answer categories: Yes, no, refusal, don’t know 

In the past 12 months, did you ... 

Additional option: in the last 4 weeks, did you ... 

a) Consult hooks, manuals, either audio tapes, videos or other documents with the intention of 
developing your joh skills? 

h) Use computer-hased software or the Internet specifically to gain joh knowledge? 

c) Observe someone perform a task with the intention of developing your job s ki lls? 

d) Take advice from colleagues at the workplace with the intention of developing your job skills? 

e) Take advice from supervisors at the workplace with the intention of developing your job s ki lls? 

f) Attend job-related conferences, trade shows, or conventions? 

If more than one activity: 

V2 What was the most important activity for developing your job skills? 

Categories from V 1 
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In CAPI or CATI, only activities with “yes ” in VI should be listed. 

V3 How much time did you spend on this activity? 

(. . .) Less than 5 hours 
(...) 5 to 10 hours 
(...) 11 to 40 hours 
(...) 41 to 160 hours 
(. . .) More than 160 hours 
(...) Refusal 
(...) Don’t know 

Comment: An open question with exact hours is not helpful here since most persons will he able to 
give hroad estimations only. As the time spent is often rather short and prohahly the first two categories 
will he answered often, categories of different lengths are chosen. 

The following question is only asked for categories a, b, c andfinVl. 

V4 Were you doing this activity . . . 

(. . .) only during paid working hours? 

(. . .) mostly during paid working hours? 

(. . .) mostly outside of paid working hours? 

(. . .) only outside paid working hours? 

(. . .) Not working at that time 
(...) Refusal 
(...) Don’t know 

Comment: Questions and answer categories on vocational informal learning are based on those in the 
AL surveys of Canada, Germany and the USA, but with some variation in detail. 



43 




EDU/WKP(2008)8 



V 5 What was the main subject of this activity? 

(Open text) 

(...) Refusal 
(...) Don’t know 

B: Informal learning of personal interest 

PI Apart from the activities we have just discussed, please tell me if you have deliberately tried to 
teach yourself anything that was a matter of personal interest -rather than for developing job skills - 
in the last 12 months by ... 

Additional option: in the last 4 weeks 

Answer categories: Yes, no, refusal, don’t know. 

(...) Learning from a family member, friend or colleague 

(. . .) Using printed material (books, professional magazines, etc.) 

(...) Using computers (online or offline) 

(...) Through television/radio/video 

(...) By guided tours of museums, historical/natural/industrial sites 
(. . .) By visiting learning centres (including libraries) 

Comment: Questions and answer categories are adapted from the EU AES, but the questions here focus on 
personal interest. 

If more than one activity: 

P2 What was the most important activity? 

Categories from PI 

In CAPI or CATI, only activities with “yes ” in PI should be listed. 

P3 How much time did you spend on this activity? 

(...) Less than 5 hours 
(...) 5 to 10 hours 
(...) 11 to 40 hours 
(...) 41 to 160 hours 
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(. . .) More than 160 hours 
(...) Refusal 
(...) Don’t know 

P4 Please list the subject you taught yourself using this method. 



(Open text) 



(...) Refusal 

(. . .) Don’t know 

Conclusions 

1) First results from the EU AES indicate that its concept of measuring informal learning causes some 
problems. Therefore, a modified module for measuring informal learning including elements from 
different country instruments is presented here. 

2) Since the proposed module only covers some aspects of informal learning, countries should he free to 
add further aspects according to national policy priorities. 

3) However, international comparison of participation in informal learning should refer to the same 
activities and not to varying instruments. 



Figure 13: 

Measuring informal learning in international AL surveys 



Do 



Develop key questions 
accepted by different countries 

Optional: Add additional 
questions according to 
policy interest 






Do not 

★ Compare participation rates 
based on short and long activity 
lists 






Include parts of 
random learning? 



► Policy interest in informal learning varies between countries, and so will the scope of 
questions addressed in an AL survey. A core set of questions used by all countries would 
be helpful. 

► International comparison should be based on the core set of questions. 

► Open questions: Include parts of random learning? Separate vocational and personal 
interest? 



Helmut Kuwan 2008, OECD AL Module 
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6. Volume and costs of Adult Learning 

6.1 Volume 

Apart from participation rates in learning activities, volume is one of the most important indicators in 
an AL survey. Since the volume of different learning activities varies considerably, the structures of adult 
education may vary with regards to the numbers of individuals participating and volume of participation. 
Both indicators together create a far more accurate picture than looking at them separately. 

When interpreting results it has to be kept in mind that the ability to remember the volume of a 
learning activity is a greater problem than that of remembering the simple fact of participation. Thus, the 
results on volume are within a wider tolerance range than participation rates. 

The statistical feasibility of a volume indicator also varies considerably between formal, non-formal 
and informal learning activities. While most experts agree that reasonably valid results can be expected for 
formal and non-formal education activities, this is far more doubtful for informal learning. The main 
problem is that respondents will not be able to recall the hours of informal activities learning during a 12 
month reference period. For this reason, the volume of informal learning is excluded from the EU AES and 
most national AL surveys, while volume aspects of formal education and non-formal education in an AL 
survey are generally accepted. 

In a shorter reference period than 12 months, the volume of informal learning might be covered. 
However, this indicator would not be too helpful because no comparison is possible with participation in 
other types of learning. Another possibility might be to cover the volume of informal learning in rather 
broad categories instead of open figures (see the suggestions in section 5.3). This would allow at least 
some comparison on an aggregate level. 

The focus is on volume during the reference period. However, the learning activity may have 
started in the year before or end in the year after the reference period or it may take longer than one year. 
In these cases the volume for the reference period has to be calculated as a share of the total volume. In 
doing so, respondents; often will need help by the interviewer. Therefore it is important to train the 
interviewers in advance how to help the participants in doing volume calculations. 

Volume will be estimated in hours. To make it easier for the respondents, measurement in terms of 
days (one day defined as 8 hours), weeks or months could also be allowed. The inclusion of homework and 
travelling time is suggested here as well, but this should be an additional question separated from the 
volume of time spent for “classroom” learning activities. 

The most important issue in measuring volume seems to be training of the interviewer. As different 
concepts in the EU AES all had their difficulties, flexibility of instruments is important. Therefore, no 
recommendation on question and answers is given here. 

Since volume data may refer to hours, days, weeks and months it is crucial for international 
comparison to develop a common understanding of recalculation. To give a concrete example: How many 
hours define a learning day? A learning day could be defined by 6, 7 or 8 hours. In the author’s opinion, 
these three options all seem to be acceptable, but it is essential to use the same definition of a learning day 
in every country. 

Conclusions 

1) Apart from participation rates in learning activities, volume is one of the most important indicators 
when measuring adult learning. However, respondents recall volume less accurately than participation. 
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2) Volume information mostly focuses on formal and non-formal education. Using broad categories 
instead of open questions might lead at least to some volume information on informal learning. 

3) Whatever the questions and answer categories, quite a few respondents will need help from the 
interviewer when calculating volume of learning activities in the reference period. Therefore, 
interviewer training on volume questions is particularly important. 

4) AL volume data may refer to hours, days, weeks or months. In international comparison, it is essential 
to use the same recalculation formula (e.g. how many hours define a learning day?). 

6.2 Costs 

There is much policy interest in the costs of education and learning, but it has to be reinforced that 
individuals are not cost experts. Learners know very little about issues like indirect subsidies, incidence of 
costs, etc. Rather complicated cost considerations may be of high political priority, but asking about these 
in a population survey will lead to results of low empirical validity. 

Therefore, information gathered on cost components has to be very simple and should strictly refer to 
the learner’s perspective instead of the perspective of educational institutions or senior economists. For this 
reason, it is recommended here to limit information to a few aspects for which valid information can be 
expected: 



• Volume of time spent for a learning activity. This should also include information on whether the 
participation was during work or not. This information can be used for developing a cost model: 

• The amount paid by the participant (for tuition, registration, exam fees, books, teaching material, 
etc.). 

According to most studies of investments in learning, the time that individuals spend in education and 
learning is by far the biggest part of individual learning costs. To obtain data on this issue, a question is 
needed on whether learning occurred during paid working time or not. Based on this information time 
volume can be monetarised in a cost model by combining expenditure of time and income of employees in 
different occupational groups, or other assumptions. 

One main advantage of a cost model is that its assumptions are transparent. It also has to be noted that 
volume estimation of individuals are much more valid than aggregated estimations from company 
experts.^’ Therefore, the data quality of a cost model based on volume data of individuals would lead to 
much better results than data based on company surveys like the Continuing Vocational Training Survey 
(CVTS). 

Beyond monetarized volume data, the scope of valid information on the costs of learning activities 
available from individuals seems to be rather limited. Therefore, it is recommended here that additional 
cost information should be focussed on one simple question: the amount contributed to a course by the 
participant. 

Most AL surveys cover two aspects of this issue: 



Case studies comparing answers of employees and company experts referring to participation and volume of 
non-formal vocational education in the same company illustrate that many company experts do not dispose 
of exact data on these issues. Particularly in big companies, the answers of company experts often refer to 
rather broad estimates that only include part of the learning activities reported by the employees. 
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• Tuition, registration, exam fees, 

• Books or technical study means. 

Nearly all AL surveys also include questions on employer support and some country surveys like 
Finland and Canada ask for other sources of financial support as well. 

The inclusion of employer support in the core AL module is recommended here while questions on 
other sources should he optional for countries. Overall, this concept follows the EU AES and includes most 
of the cost aspects covered in country surveys. 

Asking questions on cost aspects for formal and non-formal education only is proposed. 

Did any problems occur with the cost section in the EU AES? So far, participating countries did not 
report serious problems. However, some results of the German Pretest comparing results from the national 
AL survey (BSW) and the EU AES-concept for non-formal education are worth noting. 

The instruments use somewhat differing wording, a different order of questions and categorical 
questions for costs in BSW instead of exact figures in the AES, but the content covered in both surveys is 
rather similar. However, results differ considerably. Using the BSW instruments, individuals have taken a 
share of costs more often than in the EU AES concept, while employers did so less often. One reason for 
this difference might be that in the BSW the rate of non-vocational learning in non-formal education 
activities is significantly higher than in the EU AES. However, this effect alone cannot explain the big 
difference in the amount paid by individuals. The overall amount paid by the group who took a share of 
costs below EUR 50 varies between 10% in the AES and 59% in the BSW. 

These figures illustrate that the results on cost issues are highly sensitive to relatively small changes in 
a survey’s instruments. In particular, the results seem to vary between questions with cost categories and 
questions asking for exact figures. We do not know which of the results above are closer to reality. 
Therefore, our first recommendation is to do further methodological research on this issue. 

Second it is recommended here to use strictly identical instruments in international AL surveys when 
dealing with cost issues as even minor variations may lead to big differences. 

Subject to new findings from methodological experiments, our recommendation at present is to follow 
the EU AES concept with one minor modification: As Statistics Sweden suggested, the order of questions 
on costs should be changed for formal and non-formal education so that all questions on one topic (e.g. 
tuition, registration, exam fees) should be asked consecutively, followed by the question on the second 
topic. 

The following section lists the questions and answer categories proposed. “Refusal” and “don’t know” 
categories should be included but are not shown here. Neither are technical hints for filters in CAPI or 
GATE 

Questions and answer categories 
Cl Did this education take place 

(. . .) Only during paid working hours? 

(. . .) Mostly during paid working hours? 
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(...) Mostly outside paid working hours? 

(. . .) Only outside paid working hours? 

(...) Not working at that time 

C2 Did your employer or prospective employer pay in part or in full for tuition, registration, exam 
fees, regarding your studies as part of this education? 

(...) Yes, totally 

(...) Yes, partly 

(...) No, not at all 

(...) There were no such costs 

(...) Not employed at that time 

C3 Did your or any member of your family pay partly or totally for tuition, registration, exam fees, 
regarding your studies as part of this education? 

(...) Yes, totally 

(...) Yes, partly 

(...) No, not at all 

(...) There were no such costs 

C4 In the last 12 months, how much did you personally, or any member of your family, pay for 
tuition, registration, exam fees, regarding your studies as part of this education? 

EUR 

C5 Did your employer or prospective employer pay in part or in full the expenses for books or 
technical study means, regarding your studies as part of this education? 

(...) Yes, totally 

(...) Yes, partly 

(...) No, not at all 

(...) There were no such costs 
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(...) Not employed at that time 

C6 Did you or any member of your family pay partly or totally the expenses for books or technical 
study means, regarding your studies as part of this education? 

(...) Yes, totally 

(...) Yes, partly 

(...) No, not at all 

(...) There were no such costs 

C7 In the last 12 months, how much did you personally or any member of your family pay for 
books or technical study means, regarding your studies as part of this education? 

EUR 



Conclusions 

1) Questions on costs should be simple. They should strictly refer to the learner’s perspective, not to an 
institutional perspective. 

2) Learners know very little about issues like indirect subsidies, incidence of costs, etc. These aspects 
may be of high political priority but asking about them in a population survey will lead to results of 
low empirical validity. 

3) The time spent in education and learning is the biggest part of individual learning costs. Therefore, 
one key question is whether learning took place during paid working time or not. 

4) Eollowing the basic concept of the EU AES and most country surveys, we suggest focussing 
additional cost information on the amount contributed to a learning activity by the participant. This 
includes two aspects: (A) tuition, registration, exam fees; (B) books or technical study means. 

5) Cost issues seem to be highly sensitive to relatively small changes of categories and questions in a 
survey. Eor this reason, we recommend using strictly identical instruments for measuring cost aspects 
in international AL surveys. 

6) Eurther methodological research on this issue is needed to clarify which concepts lead to results that 
are closer to reality. 
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Figure 14: 

Measuring cost aspects in international AL surveys 



Do 

★ Ask simple questions on 
cost issues 




Do not 

A Ask for indirect subsidies, 
incidence of costs, etc. 




★ Refer to formal and 
non-formal learning 



A Compare results based on 
varying instruments 



► Questions on costs should be simple. 

They should refer strictly to a learner's perspective. 

► There are two key issues: 

1) Did learning take place during paid working hours? 

2) How much did participants pay? 

► Results seem to be highly sensitive to small changes of survey instruments. 




Helmut Kuwan 2008, OECD AL Module 
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7. Providers and fields of Adult Learning 

7.1 Providers 

Concepts and definitions 

As stated in chapter 5.3, informal learning takes place outside of the educational system; usually no 
professional teacher - and no provider - is involved. Formal education on the other hand varies so much 
between countries that this report recommended no generalised questions on participation in this field (see 
5.1). Therefore, this section on providers refers to non-formal education only. 

The term “provider” is used hy educational experts as well as in colloquial language. The colloquial 
use of the term is ambiguous. It may refer to the institution providing the teacher or the learning activity as 
well as to the location where learning is provided. So who should be considered to be the provider of 
training when those two aspects differ, e.g. when an AL institute provides training in a company? 

The criterion for defining a provider should be the teacher or lecturer rather than the place of learning, 
as stated in the EU AES manual. The EU AES uses the following definition: 

“The provider of education is defined as enterprise/municipality/governmental authority/private person 
who provides the teacher, lecturer or instructor for the learning activity. The place for learning activity or 
the organisation/enterprise which paid for the learning activity should consequently not be stated.” 

It is very important to communicate this concept to the respondent. The interview should be very clear 
on this issue (see “the “question and answers section” below). 

Provider structures may differ very much between countries. Some providers will be very frequent in 
one country and less important or even not existing in others. Therefore it will not be possible to develop a 
manageable list of all important national providers for the countries involved in an international survey 
simply because this list would be far too long. 

On the other hand, some providers exist in nearly all countries. These providers constitute a “core set” 
of common categories to which all countries are able to gather information. These common categories 
should be used as a basis for an international comparison of providers. However, these categories often 
will not lead to sufficient information on a national level. Therefore we suggest combining common 
and country-specific categories. 

The methods used in a survey to gather information on providers may differ between countries. 
Countries may use 

• A list of providers with common and country-specific categories, including a residual category 
for other providers or; 

• A national list of providers which is transferred to the common international categories after the 
interview or; 



EU AES manual 2005, p. 27. This definition stating that “The provider ... is defined as ... (the one) who 
provides.” also illustrates the difficulties in defining the term. Nevertheless the difference stated in the 
definition seems to be appropriate. 
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• An open question which is post-coded according to the proposed categories after the survey; 

All these methods may lead to meaningful information. The choice of the appropriate method should 
he left to countries. However, experience from some countries conducting the EU AES survey (e.g. 
Cyprus) shows that post-coding answers to an open question on providers can he quite difficult. 

If lists are used in a survey, is it better to use separate lists for providers of vocational and non- 
vocational or to use an integrated provider list? Again, this decision depends on the provider structures of 
the countries. 

If it is feasible we prefer to use an integrated list of providers for both sectors. In some countries, 
however, provider structures in these sectors may differ so much that an integrated list would be too long. 
In this case, the use of separate lists would be a better solution. Therefore, no general recommendation is 
given here. 

Questions and answer categories 

The following questions and answer categories show examples for the first and third method 
mentioned before. 

Question 1 refers to a core set of common categories to which each country should add categories for 
country-specific providers wherever this is necessary. The listed categories refer both to vocational and 
non-vocational non-formal education. They could be changed into two questions with separate lists for 
each sector if necessary. The answer categories of question 1 combine slightly modified input from the EU 
AES, Australia, Canada, Einland, Germany, Sweden and the USA. 

The second method used in question 2 is to ask an open question and to post-code the answers. In 
doing so, the question is very simple but, as stated above, the difficulties of post-coding the answers should 
not be underestimated. Question 2 is taken from the EU AES questionnaire from Cyprus to which a hint 
addressed to the interviewer were added. 

Pr 1 Who was the provider of this activity? 



Interviewer: In case of ambiguity, please explain to the respondent: 

“Provider ” is defined by the teacher or coarse leader, not by the learning location. 



(. . .) Employer, own company 

(. . .) Private training institute or private business school 

(. . .) Other company or commercial institution where training is not the main activity 
(e.g. supplier of equipment) 

(...) Employers’ organisation, chamber of commerce 
(. . .) Trade union, labor union 
(...) Professional association or organisation 
(...) University, university college, high school 
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(. . .) Non-profit association, e.g. cultural society, political party, NGOs, etc. 

(. . .) Trade/ vocational school or publicly-funded technical institute 
(...) Dance school or institute 
(. . .) Driving school 

(. . .) Private person (e.g. students or teachers giving private lessons) 

(. . .) Categories for country-specific providers (to be added by each country) 

(. . .) Other provider, please specify: 

(...) Don’t know 
(...) Refusal 

Pr 2 Who was the provider of this activity? Please give the name of the institution / service. 

Interviewer: In case of ambiguity, please explain to the respondent: 

“Provider ” is defined by the teacher, not by the learning location. 



Note: The answers to question Pr 2 will be post-coded after the interview. 



Figure 15: 

Providers of non-formal education 



Do 






★ Define "provider" by the 
teacher, not by learning location 

★ Combine a "core set" of common and 
country-specific provider categories 



Do not 

★ Leave the decision what the 
term "provider" means to the 
respondent 




► International AL surveys should gather information on a "core set" of common 
categories. In most countries, country-specific categories will have to be added. 

► The methods used in a survey may differ between countries. Countries may use 
=4= a list of providers 

=4= an open question which is post-coded. 

► Do not underestimate the difficulties of post-coding answers to an open question on 
providers ! 



Helmut Kuwan 2008, OECD AL Module 



54 




EDU/WKP(2008)8 



7.2 Fields of learning 

Concepts and definitions 

The term “fields of learning” refers to the content of learning activities. ISCED provides a 
classification system for not only levels of education and training but also for fields of education (see 
Andersson, R., Olsson A.K. 1999). Some conceptual issues on classification also were discussed in the 
CEDEEOP -project “Harmonised List of Learning Activities (HaLLA)” (see Gnahs et. al. 2002). 

The ISCED classification of fields was developed mainly for formal education. The EU AES uses this 
classification also for non-formal education and informal learning. This is a very interesting approach. 
How well did it work in practice? 

Although a few countries report problems in coding formal education activities by ISCED fields (for 
example, in the German pretest one out of four formal learning activities could either not be classified at all 
or not unambiguously) most countries had no difficulty with this classification in formal education. 

In non-formal education and informal learning, the recommendation in the EU AES was to use an 
open question with the possibility of a three digits post coding. Several major problems were connected 
with post coding here. 

In a methodological experiment from Statistics Sweden, two persons independently coded learning 
activities in non-formal education and informal learning. The reliability of the coding seriously decreases 
at ISCED digit level 3: “For non-formal learning activities, the quality is much better for field of education 
according to ISCED on 1 digit level or 2 digit level than for ‘fields of education and training’ on 3 digit 
level” (Larsson 2006, p.38). Reliability of ISCED field classification decreases particularly in non-formal 
education but also in informal learning. 



Another problem of ISCED fields is a very unequal distribution of field codes (see below). Results 
from the German Pretest show that there are four field codes at the 1 - digit level that individually include 
only 0-2 percent of learners in formal and non-formal education and in informal learning while the 
aggregated share of four other field codes varies between 45% and 63%. If distributions in other countries 
show similar results, codes will have to be revised. 

When using open questions, sometimes the descriptions of the respondent are not detailed enough for 
coding. Training of the interviewers reduces this problem to a certain extent. In order to get more 
information for coding some countries ask one question on the name of the learning activity and another 
one on the main subject. Eor example, Sweden asks for the name of the programme, course or other 
activity and for the main content of the learning activity in formal and non-formal education while only 
one question about the main content is used for post coding field of informal learning. Some other 
countries (e.g. Einland, the USA) use a similar approach. 

An alternative option would be to use a list or show cards for classification on the first digit level and 
to ask an open question on content afterwards. In this case, the learners would classify the fields of the 
learning activities. We expect a classification by the learners themselves to be more valid than an ex post 
classification of an open question by other persons. However, a list would have yet to be developed as the 
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categories recommended for the first digit of the EU AES do not appear to he a final solution for non- 
formal education and informal learning already.^^ 

Many learning activities refer to several subjects. If possible, classification should be based on the 
major field according to the rating of the learner. In addition, a separate code is needed for learning 
activities where the main focus is an interdisciplinary approach. 

Questions and answer categories 

To pin down open questions on fields in this section would be rather easy but not very helpful. The 
key issue to deal with is the classification of fields. 

The ISCED classification of fields seems to work acceptable in formal education. To use it also in 
non-formal education and informal learning is a very interesting idea, but results from the EU AES show 
serious difficulties when putting this classification into practice. 

According to the author’s opinion, there is still a need to develop a classification system of fields for 
non-formal education and informal learning. This task goes beyond the scope of this report and would need 
a separate research project to solve it. ISCED fields act as an important discussion point but probably they 
are not yet a conceptual solution. Alternative approaches, e.g. a combination of a common list for the first 
digit level followed by an open question, also should be tested. 

Although this report cannot solve the classification problem, a concrete input for discussion might be 
helpful. The following answer categories are an example for a list of subjects or fields in vocational non- 
formal education that might be applicable for vocational informal learning as well. The answer categories 
mainly come from the Canadian Workplace and Employee Survey (WES) with a few additions from the 
EU AES, CVTS and the author. 

F 1 What was the main subject or field of this course or training? 

(. . .) Orientation for new employees 
(. . .) Computer hardware or software 

(...) Managerial or supervisory training (e.g. coaching, leadership) 

(...) Office administration (accountancy, secretary, office equipment, etc.) 

(...) Manufacturing, non-office machinery and equipment (e.g. forklift, truck maintenance) 

(...) Sales and marketing (e.g. consumer service) 

(...) Occupational health and safety, environment, sustainability 

(...) Group decision-making, problem solving, team building, communication 

(...) Personal development (e.g. time management, stress management, retirement) 

(...) Personal services (tourism, security, etc.) 



For telephone interviews, a long list is not appropriate. A list would have to be adapted to the needs of a 
telephone interview, e.g. by splitting it into two or more questions 
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(. . .) Education, teacher training, etc. 

(. . .) Mathematics, physical sciences, biology, chemistry 
(. . .) Health, medicine, dentistry 
(. . .) Language training 
(. . .) Apprenticeship training 

(. . .) no single main subject, interdisciplinary approach 

(. . .) Other field, please specify: 

(...) Don’t know 



Figure 16: 

Fields of learning 



► The ISCED-classification of fields was developed for formal education. 

Most countries did not report major difficulties when classifying formal 
education by ISCED fields in EU AES. 

► To use ISCED fields also in non-formal education and informal learning is an interesting 
idea. However, in EU AES serious difficulties are reported by many countries for 
informal learning and by some for non-formal education. 

► There is still a need for developing a classification system for fields in non-formal 
education and informal learning. ISCED fields are an important input but no final 
solution. 

► Alternatives to the EU AES approach of asking an open question on fields and 
postcoding it should be tested. One alternative might be a common list for the first digit 
followed by an open question for more details on content. 



Helmut Kuwan 2008, OECD AL Module n 



57 




EDU/WKP(2008)8 



8. Obstacles and benefits of Adult Learning 



8.1 Concepts and definitions 

This section refers to obstacles and benefits of non-formal education only. However, most of the 
following comments apply for formal education and some for informal learning as well. 

The basic meaning of the terms “obstacles” and “benefits” is easy to understand. This might be the 
reason why no specific definition was found in the manuals and papers examined. Still, a short 
specification of terms might be helpful. The working definition for this section is as follows: 

Obstacles in adult learning are influencing factors reducing the probability of adults to participate in 
adult learning activities. 

Benefits of adult learning are tangible outcomes or perceived personal advantages resulting from 
participation in adult learning activities. 

AL surveys mostly gather information on both issues. In European AL surveys, the major focus often 
seems to be on obstacles while non-European countries, particularly the USA and Canada, tend to cover 
obstacles and benefits more equally. The author prefers the non-European approach here. 

It should be stated clearly that it is not possible to cover the variety of discussions on obstacles and 
benefits within a few pages in this report. Rather than summarising the discussion broadly, the aim of this 
section is to point out some general and specific “empirical basics“. These remarks refer to five issues: 

• The limits of quantitative data; 

• Limits of international surveys; 

• Social desirability; 

• The need to look at different target groups; 

• The need to connect items with behavioral aspects. 

Knowing more about obstacles and benefits of adult learning is very important for policy makers. 
Quantitative data alone, however, will not give enough information for designing programmes. With 
regard to special target groups, qualitative data are indispensable. Policy relevant studies on obstacles and 
benefits require the application of both quantitative and qualitative methods. 

There is an important difference between the potential capacity of information in national and 
international AL surveys. In international AL surveys, it is only possible to cover some rather broad 
aspects. National surveys allow for information that is more detailed. In national surveys, information can 
be tailored to the national educational framework. 

Social desirability may strongly influence questions on obstacles and benefits connected with 
participation in education and training. The majority of the population accepts the general importance of 
life-long learning. This does not necessarily mean that respondents equally approve the importance of life- 
long learning for their personal situation. Therefore, specific obstacles items connected to the personal 
situation of the respondents will lead to more valid information than general items. To illustrate this by an 
example: An item on obstacles stating “1 had no time” will lead to less valid results then specific items like 
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”I had no time because of my family responsibilities” or “1 had no time because 1 was too busy at work” 
(see question and answers for more detail). 

When looking at obstacles it is also important to differentiate between participants and non- 
participants. The EU AES concept (see AES manual 2005, p. 28) tries to focus on four groups: 

• Respondents who already participated and did not want to participate more. 

• Respondents who already participated but wanted to participate more; 

• Respondents who did not participate but wanted to participate; 

• Respondents who did not participate and did not want to participate; 

Eor our discussion the last three groups appear to be more relevant than the first one; 

There are many different ways of classifying obstacles. Erom an empirical perspective, the following 
seven aspects seem to be particularly important: 

1) General attitudes towards life-long learning; 

2) Learning dispositions (e.g. fear of failure, self-confidence); 

3) Individual preference for learning methods; 

4) Personal life situation (e.g. family situation, health problems, etc.); 

5) Learning environment (at work and private); 

6) Institutional framework (time -schedule of courses, costs, regional disparity etc.); 

7) Transparency (need for more information or consultancy on life-long learning). 

Wherever possible, a questionnaire should connect obstacles to behavioural aspects. Affirmation of 
an obstacles item by someone who made an effort to participate in non -formal education but did not 
succeed is “harder evidence” than affirmation by someone who never tried. 

These general remarks may help the reader to find orientation in the “ocean of items” connected with 
obstacles. Eurther discussion on specific items is found in the question and answer section. 

As for benefits, a variety of aspects exists, too. In addition to short-time benefits, the discussion also 
includes social outcomes of learning such as social participation, mental and physical health, 
intergenerational effects, etc. It will not be possible to take up most of these aspects in this report. The 
questions and answers stated below mainly deal with short-time output effects, not with outcome. 

Methodologically, panel studies would be particularly interesting to measure benefits of adult 
learning, but presently, international AL studies do not usually take the form of panel studies. Therefore, 
we suggest asking no questions that would require a panel design to answer them. 
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8.2 Questions and answer categories 

The following questions and answer categories refer to vocational non-formal education only. In order 
to keep the instrument rather simple, no questions on ohstacles are asked for persons who already 
participated and did not want to participate more. Also, no questions are asked on specific ohstacles that 
participants had to overcome. The questions are addressed to the three major target groups of this section: 

• Respondents who already participated hut wanted to participate more; 

• Respondents who did not participate hut wanted to participate; 

• Respondents who did not participate and did not want to participate. 

The ohstacle items lean on the EU AES questions with a few modifications of answer categories, part 
of them resulting from Einnish item testing. The suggestion helow also simplifies the splits used in the EU 
AES. 

In addition to the questions from the EU AES, question and / or answer categories from Canada, the 
USA and suggestions of the author were added. 

The questions on benefits mainly refer to questionnaires from Canada and the USA, with a few 
additions from the EU AES (particularly from Einland) and the German BSW. 

Question to non-participants in the last 12 months 

Q1 In the last 12 months, did you want to participate in education and training? 

(...) Yes Proceed to question Q3 

(. . .) No Proceed to question Q4 A 

Question to participants in the last 12 months 

Q2 Would you have liked to participate in some additional education or training during the last 12 
months? 

(...) Yes Proceed to question Q3 

(...) No Proceed to question Q6 

Q3 Did your interest lead you to actually look for any information about training, workshops, 
seminars, courses, or classes? 

(...) Yes 

(...) No 

If Q1 was yes proceed to Q4 B 
If Q2 was yes proceed to Q5 A 
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Target group for Q4 A: Non-participants who did not wish to participate 

Q4 A What were the reasons why you did not wish to participate in any education or training 
during the last 12 months? 

Target group for O 4 B: Non-participants who wished to participate 

Q4 B Do any of the following reasons explain why you did not participate in any education or 

training during the last 12 months? 







YES 


NO 


NOT WORKING 
AT THAT TIME 


a) 


You did not need it for your job 








b) 


You did not need it for your personal (not job related) reasons 








c) 


You did not have the prerequisites to be accepted to participate 








d) 


Training was too expensive for you 








e) 


Lack of employer support 








f) 


You did not have time because you were too busy at work 








g) 


You did not have time because of your family responsibilities 








h) 


There was no training offered within a reachable distance 








i) 


You were not confident with the idea of going back to 
something that is like school 








j) 


Your health or age did not allow you to participate 








k) 


You could not find the training you wanted to take 








1) 


Lack of confidence 








m) 


Was there another important reason? If yes, please specify 









If respondent marked more than one obstacle: 

Q4 C Which reason was most important? 

Repeat categories from Q4 A 

(...) Don’t know 

All respondents in Q4 B and C proceed to Q 6! 

Target group: Participants who wanted to participate more 

Q5 A Do any of the following reasons explain why you did not participate in any additional 

education or training during the last 12 months ? 
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YES 


NO 


NOT WORKING 
AT THAT TIME 


a) 


You did not have the prerequisites to be accepted to participate 








b) 


Training was too expensive for you 








c) 


Lack of employers support 








d) 


You did not have time because you were too busy at work 








e) 


You did not have time because of your family responsibilities 








f) 


There was no training offered within a reachable distance 








g) 


You were not confident with the idea of going back to something 
that is like school 








h) 


Your health or age did not allow you to participate 








i) 


You could not find the training 
you wanted to take 








j) 


Lack of confidence 








k) 


W as there another important reason? (...) 

If yes, please specify 









If respondent marked more than one obstacle: 

Q5 B Which reason was most important? 

Repeat categories from Q5 A 
(...) Don’t know 



Target group: participants and non-participants 

Q6 In the last 12 months, was there any education or training that you needed to take for a current 
or future joh hut did not? 

(...) Yes 

(...) No 

(...) Refusal 

Q7 Do you have a good overview of training which is useful for you? 

(...) Yes 
(...) No 



62 





EDU/WKP(2008)8 



(...) Refusal 

Q8 Would you like to have more information and advice on training which is useful for you? 

(...) Yes 
(...) No 
(...) Refusal 

Comment: A combined analysis of the simple questions 07 and 08 on transparency obstacles had proved 
to be very helpful in examining problems during the German reunification. 

Benefits 

B 1 How much have you used (or do you expect to use) the skills or knowledge that you acquired 
from this activity? 

(...) A lot 

(...) A fair amount 

(...) not too much 

(...) Very little 

(...) Not at all 

Comment: B 1 can be asked either as a summarized question with regard to all vocational non-formal 
education in the last year or more specifically with regard to one activity. In this case, very short activities 
(e. g. less than 8 hours) should be excluded. 
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B 2 Up to now, has the training in which you participated actually helped you to .... 

(mark all that apply) 







YES 


NO 




Do your job better? 








keep your job? 








Help you find or change a job? 








Change your occupational field? 








Increase your income? 








Get a promotion? 








Learn entirely new skills? 








Start your own business? 








Improve your self-confidence? 








Improve your self-confidence 








other important benefit, please specifv 








Refusal 








Don’t know 







If respondent marked more than one benefit: 

B 3 Which benefit was most important? 

Repeat categories from B 2 
(...) Don’t know 

B 4A Did this activity lead to a certificate or license which is required for your current or planned 
job or profession? 

(...) Yes Proceed to question B 4B 
(...) No End of questions on benefits 
(...) Don’t know 

B 4B What kind of a certificate or license did this activity lead to? 
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Comment: Add country-specific codes 

B 5 Did you receive this certificate or license? 

(...) Yes 
(...) No 

(. . .) Not finished yet 
(...) Refusal 
(...) Don’t know 



Figure 17: 

Obstacles and benefits of Adult Learning 



Dd y 


Da not 


* Use specific items (e g. "no time^^ 


* Use very general items 'iL 


because of my family responsibilmes') 


(e g. "I had no time") ^ 


* Tiy to conne ct obstac les with behavior 


* Use items with strong social 


instead of asking for attitudes only 


desirability (e.g. "I am too lazy") 


* Ask for bene fits as well 





► Ite ms on obstacles should be taUore d to spe cific situations . 



► Obstacle ite ms connected to real behavior will lead to mo re meaningful infoimation than 
questions on attitudes. 

► Infoimation should be gathe redonbenefitsas well. Questions on benefits should no t be 
restricted to economic benefits only 



Helnut Kuwan 2008, OECD AL Module 
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9. Background variables 

The focus of this section is to state which background variables should be included in an international 
AL survey. Quite a few of these variables are more or less self-explanatory (sex, year of birth, etc.), but 
discussions on some items are controversial. 

The first part of this section lists the background variables that should be included in an international 
AL survey. The second section deals with an example of a controversial issue: the occupational status one 
year before the survey as recommended by the EU AES. 

9.1 Overview of background variables 

We consider the following background variables as very important for an AL survey: 

• Sex 

• Year of birth 

• Nationality 

• Years of residence in this country 

• Country of birth 

• Degree of urbanisation"^® 

• Number of persons living in the same household (including the respondent) 

• Highest level of education or training successfully completed 

• Eield of highest level of education or training successfully completed 

• Main current labour status 

• If possible: Labour status at the time of the learning activity (see 9.2) 

• Professional status; occupation 

• Economic sector 

• Economic activity of the local unit 

• Number of persons working at the local unit 

• Permanency of the job 

• Pull-time / part-time distinction 

^ This information is usually taken from AR -registers or from sample-points in RR-samples. 
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• Existence of more than one job or business (currently) 

• Income group based on the monthly (take home) pay from main job 

9.2 A controversial example: Professional situation one year before a survey 

Among the background variables recommended with highest priority for the EU AES is the 
professional situation of the respondent one year before the survey. This includes a few variables like 
professional status, occupation, permanency of job etc. How much useful information is gained from these 
variables? 

The intention of the EU AES is to connect these variables with learning activities. However, the 
occupational status one year before the survey may differ from the one at the time of the interview. In fact, 
the link between these variables and learning activities is not too close. Capturing at least some information 
on the labour situation at the time of the learning activity will raise the validity of data on the link between 
occupation and learning. Therefore, the German AES decided to ask for information on occupational 
events instead of the occupational status one year before the survey. The Canadian AETS goes one step 
further and asks for the labour status at the time of the learning activity. 

Although it will not be possible to cover a one-year job history of respondents in detail in an AL 
survey, it is possible to get some valid information with little response burden. Eor example, the simple 
addition of the category “not employed at this time” to a question on employer’s cost share would bring on 
a closer connection already as this answer reveals the occupational situation at participation time (see 
question C2 in section 6.2). 

Questions referring to events in professional life will lead to information connected closer to 
participation in learning activities than the occupational status one year before the survey. This approach 
causes a rather small response burden and allows capturing the occupational situation at the time of the 
learning activity for quite a few groups. 

When asking directly for the labour status at the time of the learning activity this information will be 
available for all groups of learners. However, as this question will refer to several learning activities the 
response burden is also somewhat higher. Methodologically, this approach is recommended if the survey 
scope allows its use. 

Questions and answer categories 

The following examples on questions and answers refer to two different approaches: 

• Information on occupational events (Bl); 

• Information on the labour status at the time of the learning activity (B2 to B 4). 

The categories of the following example question on events come from the ALL survey, from Canada 
and the German AES. 

B 1 During the last 12 months, were you ... 

(. . .) always working at the same job or 

(. . .) did you change your job, but were never unemployed or 
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(. . .) did you change your job, and were unemployed for some time or 
(...) were you unemployed all the time or 
(. . .) did you retire during the year? 

(...) Refusal 
(...) Don’t know 

The following questions and answers are taken from the Canadian AETS with a slight condensation. 

B 2 While you were taking this program/course last year, were you... 

(...) self-employed? 

(...) an employee? 

(. . .) temporarily laid off? 

(. . .) not working and looking for work? 

(. . .) not working and not looking for work? 

(...) Refusal 
(...) Don’t know 

If respondent was not working at time of training, go to next block. If respondent was working while 
training and is currently unemployed, go to B4Else go to B3 

B 3 Is your current joh the same as the one you held while taking this program/course? 

(...) Yes 
(...) No 
(...) Refusal 
(...) Don’t know 
Go to next block 

B 4 Is your most recent joh the same as the one you held while taking this program/course? 

(...) Yes 
(...) No 
(...) Refusal 
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(...) Don’t know 
Conclusions 

1) We do not recommend to ask for the professional situation one year before the survey because this 
information is not linked very closely to learning activities. 

2) Event-orientated questions seem to be a better solution. 

3) Methodologically, the best solution would be to ask directly for the labour status at the time of the 
learning activity. However, the response burden of this alternative is also somewhat higher. 
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10. Final note: A word on imputation in the AL field 

Since this report deals with international comparison, some aspects heyond survey scope are very 
important. One is imputation. 

Often, analyses in international comparisons work with imputed data. This may improve data quality 
hut it may also lead to serious hazards. 

Only if very stable correlations exist will imputation lead to useful additional information. In this 
case, different countries have to use common rules and imputation models. 

Two more aspects are crucial: the use of imputation models should remain transparent, and it should 
he possible for different groups of researchers to check these models. Transparency is also needed with 
regard to other aspects: 

• Datasets have to indicate whether imputation was applied for a variable or not; 

• Data should make it possible to analyse data before and after imputation. 



So far, acceptance of these principles is far from evident for international comparisons in the AL field. 
It is recommended that imputation be used very restrictively and that it be combined consistently with 
very straightforward documentation. 



Figure 18: 

Prospects and hazards of imputation 



Do 

★ Use imputation only 
when referring to proven 
correlation 




Do not 

★ Keep imputation 
models secret 




► Imputation should be used restrictively. Only if very stable correlations exist will 
imputation increase options for analyses. 

► Imputation without common standards or without documentation of models increases 
imponderability of analyses. Datasets should include variables before and after 
imputation. 



Helmut Kuwan 2008, OECD AL Module 
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ANNEX 1: ACRONYMS USED IN THIS REPORT 



AE Adult Education 

AL Adult Learning 

ALL Adult Literacy and Lifeskills Survey 

AES Adult Education Survey 

AETS Adult Education and Training Survey 

AR Address Random 

BSW Berichtssystem Weiterbildung (Germany’s Reporting System on Continuing 
Education) 

CAPI Computer Assisted Personal Interview 

CATI Computer Assisted Telephone Interview 

CET Continuing Education and Training 

CL A Classification of Learning Activities 

CVTS Continuing Vocational Training Survey 

EU AES European Adult Education Survey 

E2f Eace-to-face 

HaLLA Harmonised List of Learning Activities 

lALS International Adult Literacy Survey 

ICT Information and Communication Technology 

IPEDS Integrated Postsecondary Education Data System 

ISCED International Standard Classification of Education 

ISSPInternational Social Survey Programme 

EES Labour Eorce Survey 

ELL Life-long Learning 

NEQNational Eramework of Qualifications 

NPSAS National Postsecondary Student Aid Study 

OECD Organisation for Economic Co-operation and Development 

PIAAC Programme for the International Assessment of Adult Competencies 

RR Random Route 

SLA Single Learning Activities 

WES Canadian Workplace and Employee Survey 
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